Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models

Sultan Alrashed, Dmitrii Khizbullin, David R. Pugh
11/10/2024
cs.CLcs.AI

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

As large language models (LLMs) grow and develop, so do their data demands. This is especially true for multilingual LLMs, where the scarcity of high-quality and readily available data online has led to a multitude of synthetic dataset generation approaches. A key technique in this space is machine translation (MT), where high-quality English text is adapted to a target, comparatively low-resource language. This report introduces FineWeb-Edu-Ar, a machine-translated version of the exceedingly popular (deduplicated) FineWeb-Edu dataset from HuggingFace. To the best of our knowledge, FineWeb-Edu-Ar is the largest publicly available machine-translated Arabic dataset out there, with its size of 202B tokens of an Arabic-trained tokenizer.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

ALLaM: Large Language Models for Arabic and English

85% match2024

Alexandria: A Multi-Domain Dialectal Arabic Machine Translation Dataset for Culturally Inclusive and Linguistically Diverse LLMs

85% match2026

Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis

84% match2024

Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM

84% match2025

Large Language Models and Arabic Content: A Review

84% match2025

101 Billion Arabic Words Dataset

84% match2024

Cite this paper

@article{alrashed2024finewebeduar,
  title  = {Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models},
  author = {Sultan Alrashed and Dmitrii Khizbullin and David R. Pugh},
  year   = {2024},
  eprint = {2411.06402},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.06402v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.06402.svg)](https://codeofpaper.com/paper/2411.06402)

Discussion