The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling

Joey Öhman, Severine Verlinden, Ariel Ekgren, Amaru Cuba Gyllensten, Tim Isbister +3 more
3/30/2023
cs.CLcs.AI

No confident code implementation found for this paper.

Abstract

Pre-training Large Language Models (LLMs) require massive amounts of text data, and the performance of the LLMs typically correlates with the scale and quality of the datasets. This means that it may be challenging to build LLMs for smaller languages such as Nordic ones, where the availability of text corpora is limited. In order to facilitate the development of the LLMS in the Nordic languages, we curate a high-quality dataset consisting of 1.2TB of text, in all of the major North Germanic languages (Danish, Icelandic, Norwegian, and Swedish), as well as some high-quality English data. This paper details our considerations and processes for collecting, cleaning, and filtering the dataset.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{hman2023nordic,
  title  = {The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling},
  author = {Joey Öhman and Severine Verlinden and Ariel Ekgren and Amaru Cuba Gyllensten and Tim Isbister and Evangelia Gogoulou and Fredrik Carlsson and Magnus Sahlgren},
  year   = {2023},
  eprint = {2303.17183},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.17183v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.17183.svg)](https://codeofpaper.com/paper/2303.17183)

Discussion