Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization

Salman Elgamal, Ossama Obeid, Tameem Kabbani, Go Inoue, Nizar Habash
6/9/2024
cs.CL

No official code found. Best implementation: CAMeL-Lab/wild_diacritics (4 stars, CC-BY-SA-4.0, last commit 2024-08-09).

Abstract

The widespread absence of diacritical marks in Arabic text poses a significant challenge for Arabic natural language processing (NLP). This paper explores instances of naturally occurring diacritics, referred to as "diacritics in the wild," to unveil patterns and latent information across six diverse genres: news articles, novels, children's books, poetry, political documents, and ChatGPT outputs. We present a new annotated dataset that maps real-world partially diacritized words to their maximal full diacritization in context. Additionally, we propose extensions to the analyze-and-disambiguate approach in Arabic NLP to leverage these diacritics, resulting in notable improvements. Our contributions encompass a thorough analysis, valuable datasets, and an extended diacritization algorithm. We release our code and datasets as open source.

View on arXivView PDF

Code Implementations(1)

CAMeL-Lab/wild_diacriticsCommunity100%
CC-BY-SA-4.0

Wild Diacritics paper repo.

42Shell, PythonJun 4, 20242 years agoCC-BY-SA-4.0

Similar Papers

Sadeed: Advancing Arabic Diacritization Through Small Language Model

86% match2025

Proper Noun Diacritization for Arabic Wikipedia: A Benchmark Dataset

85% match2025

Arabic Text Diacritization In The Age Of Transfer Learning: Token Classification Is All You Need

83% match2024

A Context-Contrastive Inference Approach To Partial Diacritization

83% match2024

Are LLMs Good Text Diacritizers? An Arabic and Yorùbá Case Study

82% match2025

More Data, Fewer Diacritics: Scaling Arabic TTS

80% match2026

Cite this paper

@article{elgamal2024arabic,
  title  = {Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization},
  author = {Salman Elgamal and Ossama Obeid and Tameem Kabbani and Go Inoue and Nizar Habash},
  year   = {2024},
  eprint = {2406.05760},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.05760v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.05760.svg)](https://codeofpaper.com/paper/2406.05760)

Discussion