MaLA-500: Massive Language Adaptation of Large Language Models

Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, Hinrich Schütze
1/24/2024
cs.CL

No official code found. Best implementation: MaLA-LM/mala-500 (5 stars, last commit 2024-04-24).

Abstract

Large language models (LLMs) have advanced the state of the art in natural language processing. However, their predominant design for English or a limited set of languages creates a substantial gap in their effectiveness for low-resource languages. To bridge this gap, we introduce MaLA-500, a novel large language model designed to cover an extensive range of 534 languages. To train MaLA-500, we employ vocabulary extension and continued pretraining on LLaMA 2 with Glot500-c. Our intrinsic evaluation demonstrates that MaLA-500 is better at predicting the given texts of low-resource languages than existing multilingual LLMs. Moreover, the extrinsic evaluation of in-context learning shows that MaLA-500 outperforms previous LLMs on SIB200 and Taxi1500 by a significant margin, i.e., 11.68% and 4.82% marco-average accuracy across languages. We release MaLA-500 at https://huggingface.co/MaLA-LM

View on arXivView PDF

Code Implementations(1)

MaLA-LM/mala-500Community66%

MaLA-500: Massive Language Adaptation of Large Language Models

50Shell, PythonFeb 21, 20242 years ago
large-language-modelsmultilingual-nlp

Similar Papers

EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

91% match1 repo32024

Large Language Models: A Survey

87% match2024

Efficient Continual Pre-training of LLMs for Low-resource Languages

87% match1 repo2024

Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data

86% match2025

LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language

86% match2 repos222024

Exploring Design Choices for Building Language-Specific LLMs

86% match1 repo42024

Cite this paper

@article{lin2024mala,
  title  = {MaLA-500: Massive Language Adaptation of Large Language Models},
  author = {Peiqin Lin and Shaoxiong Ji and Jörg Tiedemann and André F. T. Martins and Hinrich Schütze},
  year   = {2024},
  eprint = {2401.13303},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.13303v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.13303.svg)](https://codeofpaper.com/paper/2401.13303)

Discussion