CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán +2 more
11/1/2019
cs.CLcs.IRcs.LG

Official code: facebookresearch/cc_net (1,045 stars, MIT, last commit 2023-04-25).

Abstract

Pre-training text representations have led to significant improvements in many areas of natural language processing. The quality of these models benefits greatly from the size of the pretraining corpora as long as its quality is preserved. In this paper, we describe an automatic pipeline to extract massive high-quality monolingual datasets from Common Crawl for a variety of languages. Our pipeline follows the data processing introduced in fastText (Mikolov et al., 2017; Grave et al., 2018), that deduplicates documents and identifies their language. We augment this pipeline with a filtering step to select documents that are close to high quality corpora like Wikipedia.

View on arXivView PDF

Code Implementations(1)

facebookresearch/cc_netOfficial100%
MIT

Tools to download and cleanup Common Crawl data

1,045152Python, MakefileOct 29, 20193 years agoMIT

Similar Papers

The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling

86% match2023

CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages

85% match2023

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

85% match1 repo2472025

CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data

84% match1 repo252023

ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval

84% match1 repo242023

PersianLLaMA: Towards Building First Persian Large Language Model

84% match2023

Cite this paper

@article{wenzek2019ccnet,
  title  = {CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data},
  author = {Guillaume Wenzek and Marie-Anne Lachaux and Alexis Conneau and Vishrav Chaudhary and Francisco Guzmán and Armand Joulin and Edouard Grave},
  year   = {2019},
  eprint = {1911.00359},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1911.00359}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1911.00359.svg)](https://codeofpaper.com/paper/1911.00359)

Discussion