Orion-14B: Open-source Multilingual Large Language Models. (arXiv:2401.12246v1 [cs.CL])

In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts i...