PALI: A Language Identification Benchmark for Perso-Arabic Scripts

Sina Ahmadi, Milind Agarwal, Antonios Anastasopoulos
4/3/2023
cs.CL

No official code found. Best implementation: sinaahmadi/PersoArabicLID (11 stars, MIT, last commit 2023-07-11).

Abstract

The Perso-Arabic scripts are a family of scripts that are widely adopted and used by various linguistic communities around the globe. Identifying various languages using such scripts is crucial to language technologies and challenging in low-resource setups. As such, this paper sheds light on the challenges of detecting languages using Perso-Arabic scripts, especially in bilingual communities where ``unconventional'' writing is practiced. To address this, we use a set of supervised techniques to classify sentences into their languages. Building on these, we also propose a hierarchical model that targets clusters of languages that are more often confused by the classifiers. Our experiment results indicate the effectiveness of our solutions.

View on arXivView PDF

Code Implementations(1)

sinaahmadi/PersoArabicLIDCommunity100%
MIT

PALI: Language identification for Perso-Arabic Scripts

110Shell, PythonMar 25, 20233 years agoMIT
arabicbalochibrahuigilakigorani+15 more

Similar Papers

Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities

81% match2023

GlotLID: Language Identification for Low-Resource Languages

81% match2 repos2222023

GlotScript: A Resource and Tool for Low Resource Writing System Identification

81% match1 repo242023

Classification of Human- and AI-Generated Texts for English, French, German, and Spanish

81% match2023

Fumbling in Babel: An Investigation into ChatGPT's Language Identification Ability

81% match2023

Optical Script Identification for multi-lingual Indic-script

80% match2023

Cite this paper

@article{ahmadi2023pali,
  title  = {PALI: A Language Identification Benchmark for Perso-Arabic Scripts},
  author = {Sina Ahmadi and Milind Agarwal and Antonios Anastasopoulos},
  year   = {2023},
  eprint = {2304.01322},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.01322v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.01322.svg)](https://codeofpaper.com/paper/2304.01322)

Discussion