A New Massive Multilingual Dataset for High-Performance Language Technologies

Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde +8 more
3/20/2024
cs.CL

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ~5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)

95% match1 repo12025

HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

89% match2025

SMOL: Professionally translated parallel data for 115 under-represented languages

85% match2025

Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data

85% match2025

DocHPLT: A Massively Multilingual Document-Level Translation Dataset

84% match2025

Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers

84% match1 repo2192025

Cite this paper

@article{gibert2024massive,
  title  = {A New Massive Multilingual Dataset for High-Performance Language Technologies},
  author = {Ona de Gibert and Graeme Nail and Nikolay Arefyev and Marta Bañón and Jelmer van der Linde and Shaoxiong Ji and Jaume Zaragoza-Bernabeu and Mikko Aulamo and Gema Ramírez-Sánchez and Andrey Kutuzov and Sampo Pyysalo and Stephan Oepen and Jörg Tiedemann},
  year   = {2024},
  eprint = {2403.14009},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.14009v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.14009.svg)](https://codeofpaper.com/paper/2403.14009)

Discussion