AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing

Asaad Alghamdi, Xinyu Duan, Wei Jiang, Zhenhai Wang, Yimeng Wu +7 more
6/11/2023
cs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP). In this work, we present AraMUS, the largest Arabic PLM with 11B parameters trained on 529GB of high-quality Arabic textual data. AraMUS achieves state-of-the-art performances on a diverse set of Arabic classification and generative tasks. Moreover, AraMUS shows impressive few-shot learning abilities compared with the best existing Arabic PLMs.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

89% match2023

ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning

86% match2023

PersianLLaMA: Towards Building First Persian Large Language Model

85% match2023

A Survey of Large Language Models

85% match1 repo12.2k2023

KoBigBird-large: Transformation of Transformer for Korean Language Understanding

85% match2023

Large Language Models: A Survey

84% match2024

Cite this paper

@article{alghamdi2023aramus,
  title  = {AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing},
  author = {Asaad Alghamdi and Xinyu Duan and Wei Jiang and Zhenhai Wang and Yimeng Wu and Qingrong Xia and Zhefeng Wang and Yi Zheng and Mehdi Rezagholizadeh and Baoxing Huai and Peilun Cheng and Abbas Ghaddar},
  year   = {2023},
  eprint = {2306.06800},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.06800v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.06800.svg)](https://codeofpaper.com/paper/2306.06800)

Discussion