Zyda-2: a 5 Trillion Token High-Quality Dataset

Yury Tokpanov, Paolo Glorioso, Quentin Anthony, Beren Millidge
11/9/2024
cs.CLcs.AI

No confident code implementation found for this paper.

Abstract

In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-the-art for their weight class. We build Zyda-2 by collating high-quality open-source tokens such as FineWeb and DCLM, then distilling them to the highest-quality subset via cross-deduplication and model-based quality filtering. Zyda-2 is released under a permissive open license, and is available at https://huggingface.co/datasets/Zyphra/Zyda-2

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Zyda: A 1.3T Dataset for Open Language Modeling

91% match1 repo442024

The Falcon Series of Open Language Models

84% match2023

1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data

84% match1 repo3422024

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

84% match2023

Data, Data Everywhere: A Guide for Pretraining Dataset Construction

83% match2024

Training Data for Large Language Model

83% match2024

Cite this paper

@article{tokpanov2024zyda,
  title  = {Zyda-2: a 5 Trillion Token High-Quality Dataset},
  author = {Yury Tokpanov and Paolo Glorioso and Quentin Anthony and Beren Millidge},
  year   = {2024},
  eprint = {2411.06068},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.06068v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.06068.svg)](https://codeofpaper.com/paper/2411.06068)

Discussion