Exploiting Dialect Identification in Automatic Dialectal Text Normalization

Bashar Alhafni, Sarah Al-Towaity, Ziyad Fawzy, Fatema Nassar, Fadhl Eryani +2 more
7/3/2024
cs.CL

No official code found. Best implementation: CAMeL-Lab/codafication (3 stars, MIT, last commit 2024-07-06).

Abstract

Dialectal Arabic is the primary spoken language used by native Arabic speakers in daily communication. The rise of social media platforms has notably expanded its use as a written language. However, Arabic dialects do not have standard orthographies. This, combined with the inherent noise in user-generated content on social media, presents a major challenge to NLP applications dealing with Dialectal Arabic. In this paper, we explore and report on the task of CODAfication, which aims to normalize Dialectal Arabic into the Conventional Orthography for Dialectal Arabic (CODA). We work with a unique parallel corpus of multiple Arabic dialects focusing on five major city dialects. We benchmark newly developed pretrained sequence-to-sequence models on the task of CODAfication. We further show that using dialect identification information improves the performance across all dialects. We make our code, data, and pretrained models publicly available.

View on arXivView PDF

Code Implementations(1)

CAMeL-Lab/codaficationCommunity67%
MIT

Code, models, and data for "Exploiting Dialect Identification in Automatic Dialectal Text Normalization". ArabicNLP 2024, ACL.

31Shell, PythonJun 19, 20242 years agoMIT
arabicarabic-nlpdeep-learningnlptext-normalization

Similar Papers

Arabic Dialect Classification using RNNs, Transformers, and Large Language Models: A Comparative Analysis

86% match2025

NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task

85% match2024

AlcLaM: Arabic Dialectal Language Model

85% match1 repo82024

Exploring Data and Parameter Efficient Strategies for Arabic Dialect Identifications

85% match2025

Revisiting Common Assumptions about Arabic Dialects in NLP

85% match2025

Computational Linguistics Meets Libyan Dialect: A Study on Dialect Identification

85% match2025

Cite this paper

@article{alhafni2024exploiting,
  title  = {Exploiting Dialect Identification in Automatic Dialectal Text Normalization},
  author = {Bashar Alhafni and Sarah Al-Towaity and Ziyad Fawzy and Fatema Nassar and Fadhl Eryani and Houda Bouamor and Nizar Habash},
  year   = {2024},
  eprint = {2407.03020},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.03020v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.03020.svg)](https://codeofpaper.com/paper/2407.03020)

Discussion