An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)

Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón +30 more
3/13/2025
cs.CL

No official code found. Best implementation: hplt-project/HPLT-textpipes (1 star, MIT, last commit 2026-04-19).

Abstract

Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

View on arXivView PDF

Code Implementations(1)

hplt-project/HPLT-textpipesCommunity100%
MIT

Step-by-step schematic description of data processing in HPLT

11HTML, ShellDec 5, 20226 months agoMIT

Similar Papers

A New Massive Multilingual Dataset for High-Performance Language Technologies

95% match2024

HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

89% match2025

Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data

86% match2025

Investigating the translation capabilities of Large Language Models trained on parallel data only

84% match1 repo62024

DocHPLT: A Massively Multilingual Document-Level Translation Dataset

84% match2025

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

84% match1 repo2472025

Cite this paper

@article{burchell2025expanded,
  title  = {An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)},
  author = {Laurie Burchell and Ona de Gibert and Nikolay Arefyev and Mikko Aulamo and Marta Bañón and Pinzhen Chen and Mariia Fedorova and Liane Guillou and Barry Haddow and Jan Hajič and Jindřich Helcl and Erik Henriksson and Mateusz Klimaszewski and Ville Komulainen and Andrey Kutuzov and Joona Kytöniemi and Veronika Laippala and Petter Mæhlum and Bhavitvya Malik and Farrokh Mehryary and Vladislav Mikhailov and Nikita Moghe and Amanda Myntti and Dayyán O'Brien and Stephan Oepen and Proyag Pal and Jousia Piha and Sampo Pyysalo and Gema Ramírez-Sánchez and David Samuel and Pavel Stepachev and Jörg Tiedemann and Dušan Variš and Tereza Vojtěchová and Jaume Zaragoza-Bernabeu},
  year   = {2025},
  eprint = {2503.10267},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.10267v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.10267.svg)](https://codeofpaper.com/paper/2503.10267)

Discussion