EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

Shaoxiong Ji, Zihao Li, Jaakko Paavola, Peiqin Lin, Pinzhen Chen +5 more
9/26/2024
cs.CL

No official code found. Best implementation: MaLA-LM/emma-500 (3 stars, last commit 2025-12-04).

Abstract

In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability. We release the MaLA corpus, EMMA-500 model weights, scripts, and model generations.

View on arXivView PDF

Code Implementations(1)

MaLA-LM/emma-500Community100%

EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

30Shell, PythonSep 14, 202410 months ago
large-language-modelsmultilingual

Similar Papers

Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data

92% match2025

MaLA-500: Massive Language Adaptation of Large Language Models

91% match1 repo52024

MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting

88% match2024

LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language

87% match2 repos222024

LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages

87% match1 repo762024

Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement

86% match2024

Cite this paper

@article{ji2024emma,
  title  = {EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models},
  author = {Shaoxiong Ji and Zihao Li and Jaakko Paavola and Peiqin Lin and Pinzhen Chen and Dayyán O'Brien and Hengyu Luo and Hinrich Schütze and Jörg Tiedemann and Barry Haddow},
  year   = {2024},
  eprint = {2409.17892},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.17892v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.17892.svg)](https://codeofpaper.com/paper/2409.17892)

Discussion