Orion-14B: Open-source Multilingual Large Language Models

Du Chen, Yi Huang, Xiaopu Li, Yongqiang Li, Yongqiang Liu +5 more
1/20/2024
cs.CLcs.LG

Official code: OrionStarAI/Orion (811 stars, Apache-2.0, last commit 2024-06-03).

Abstract

In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts in English, Chinese, Japanese, Korean, and other languages. Additionally, we fine-tuned a series of models tailored for conversational applications and other specific use cases. Our evaluation results demonstrate that Orion-14B achieves state-of-the-art performance across a broad spectrum of tasks. We make the Orion-14B model family and its associated code publicly accessible https://github.com/OrionStarAI/Orion, aiming to inspire future research and practical applications in the field.

View on arXivView PDF

Code Implementations(1)

OrionStarAI/OrionOfficial100%
Apache-2.0

Orion-14B is a family of models includes a 14B foundation LLM, and a series of models: a chat model, a long context model, a quantized model, a RAG fine-tuned model, and an Agent fine-tuned model. Orion-14B 系列模型包括一个具有140亿参数的多语言基座大模型以及一系列相关的衍生模型，包括对话模型，长文本模型，量化模型，RAG微调模型，Agent微调模型等。

81159PythonJan 17, 20242 years agoApache-2.0

Similar Papers

Nemotron-4 15B Technical Report

86% match2024

LOLA -- An Open-Source Massively Multilingual Large Language Model

85% match1 repo112024

FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data

84% match1 repo102024

Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers

84% match2024

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

84% match2026

A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

84% match1 repo252024

Cite this paper

@article{chen2024orionb,
  title  = {Orion-14B: Open-source Multilingual Large Language Models},
  author = {Du Chen and Yi Huang and Xiaopu Li and Yongqiang Li and Yongqiang Liu and Haihui Pan and Leichao Xu and Dacheng Zhang and Zhipeng Zhang and Kun Han},
  year   = {2024},
  eprint = {2401.12246},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.12246v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.12246.svg)](https://codeofpaper.com/paper/2401.12246)

Discussion