Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models

Zaid Alyafeai, Maged S. Alshaibani, Badr AlKhamissi, Hamzah Luqman, Ebrahim Alareqi +1 more
6/28/2023
cs.CL

Official code: ARBML/Taqyim (19 stars, MIT, last commit 2024-02-13).

Abstract

Large language models (LLMs) have demonstrated impressive performance on various downstream tasks without requiring fine-tuning, including ChatGPT, a chat-based model built on top of LLMs such as GPT-3.5 and GPT-4. Despite having a lower training proportion compared to English, these models also exhibit remarkable capabilities in other languages. In this study, we assess the performance of GPT-3.5 and GPT-4 models on seven distinct Arabic NLP tasks: sentiment analysis, translation, transliteration, paraphrasing, part of speech tagging, summarization, and diacritization. Our findings reveal that GPT-4 outperforms GPT-3.5 on five out of the seven tasks. Furthermore, we conduct an extensive analysis of the sentiment analysis task, providing insights into how LLMs achieve exceptional results on a challenging dialectal dataset. Additionally, we introduce a new Python interface https://github.com/ARBML/Taqyim that facilitates the evaluation of these tasks effortlessly.

View on arXivView PDF

Code Implementations(1)

ARBML/TaqyimOfficial100%
MIT

Python intefrace for evaluation on chatgpt models

194Python, Jupyter NotebookMay 27, 20232 years agoMIT

Similar Papers

Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction

89% match2023

NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian

89% match1 repo22023

BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP

89% match2023

Language Model Behavior: A Comprehensive Survey

88% match1 repo32023

Document-Level Machine Translation with Large Language Models

88% match1 repo1012023

LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

88% match1 repo8222023

Cite this paper

@article{alyafeai2023taqyim,
  title  = {Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models},
  author = {Zaid Alyafeai and Maged S. Alshaibani and Badr AlKhamissi and Hamzah Luqman and Ebrahim Alareqi and Ali Fadel},
  year   = {2023},
  eprint = {2306.16322},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.16322v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.16322.svg)](https://codeofpaper.com/paper/2306.16322)

Discussion