Small Languages, Big Models: A Study of Continual Training on Languages of Norway

David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier +2 more
12/9/2024
cs.CL

No confident code implementation found for this paper.

Abstract

Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Northern Sámi. To address this issue, we present a novel three-stage continual training approach that substantially improves the downstream performance together with the inference efficiency for the target languages. Based on our findings, we train, evaluate, and openly release a new generative language model for Norwegian Bokmål, Nynorsk, and Northern Sámi with 11.4 billion parameters: NorMistral-11B.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{samuel2024small,
  title  = {Small Languages, Big Models: A Study of Continual Training on Languages of Norway},
  author = {David Samuel and Vladislav Mikhailov and Erik Velldal and Lilja Øvrelid and Lucas Georges Gabriel Charpentier and Andrey Kutuzov and Stephan Oepen},
  year   = {2024},
  eprint = {2412.06484},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.06484v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.06484.svg)](https://codeofpaper.com/paper/2412.06484)

Discussion