On the importance of Data Scale in Pretraining Arabic Language Models

Abbas Ghaddar, Philippe Langlais, Mehdi Rezagholizadeh, Boxing Chen
1/15/2024
cs.CL

Official code: huawei-noah/Pretrained-Language-Model (3,162 stars, last commit 2024-01-22).

Abstract

Pretraining monolingual language models have been proven to be vital for performance in Arabic Natural Language Processing (NLP) tasks. In this paper, we conduct a comprehensive study on the role of data in Arabic Pretrained Language Models (PLMs). More precisely, we reassess the performance of a suite of state-of-the-art Arabic PLMs by retraining them on massive-scale, high-quality Arabic corpora. We have significantly improved the performance of the leading Arabic encoder-only BERT-base and encoder-decoder T5-base models on the ALUE and ORCA leaderboards, thereby reporting state-of-the-art results in their respective model categories. In addition, our analysis strongly suggests that pretraining data by far is the primary contributor to performance, surpassing other factors. Our models and source code are publicly available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/JABER-PyTorch.

View on arXivView PDF

Code Implementations(1)

huawei-noah/Pretrained-Language-ModelOfficial100%

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

3,162640C++, ShellDec 2, 20192 years ago
knowledge-distillationlarge-scale-distributedmodel-compressionpretrained-modelsquantization

Similar Papers

Large Language Models and Arabic Content: A Review

88% match2025

AlcLaM: Arabic Dialectal Language Model

88% match1 repo82024

GemmAr: Enhancing LLMs Through Arabic Instruction-Tuning

87% match2024

ALLaM: Large Language Models for Arabic and English

86% match2024

A Survey of Large Language Models for Arabic Language and its Dialects

86% match1 repo62024

Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM

86% match2025

Cite this paper

@article{ghaddar2024importance,
  title  = {On the importance of Data Scale in Pretraining Arabic Language Models},
  author = {Abbas Ghaddar and Philippe Langlais and Mehdi Rezagholizadeh and Boxing Chen},
  year   = {2024},
  eprint = {2401.07760},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.07760v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.07760.svg)](https://codeofpaper.com/paper/2401.07760)

Discussion