A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs (OpenAI, Mistral, Anthropic)

Hicham Assoudi
5/5/2025
cs.CL

No official code found. Best implementation: assoudi-typica-ai/darija-toxicity-benchmark (0 stars, last commit 2025-05-04).

Abstract

This paper presents a comparative benchmark evaluating the performance of Typica.ai's custom Moroccan Darija toxicity detection model against major LLM-based moderation APIs: OpenAI (omni-moderation-latest), Mistral (mistral-moderation-latest), and Anthropic Claude (claude-3-haiku-20240307). We focus on culturally grounded toxic content, including implicit insults, sarcasm, and culturally specific aggression often overlooked by general-purpose systems. Using a balanced test set derived from the OMCD_Typica.ai_Mix dataset, we report precision, recall, F1-score, and accuracy, offering insights into challenges and opportunities for moderation in underrepresented languages. Our results highlight Typica.ai's superior performance, underlining the importance of culturally adapted models for reliable content moderation.

View on arXivView PDF

Code Implementations(1)

assoudi-typica-ai/darija-toxicity-benchmarkCommunity100%

Reproducibility materials and evaluation notebook for the paper: A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs

00Jupyter NotebookMay 4, 20251 years agoNOASSERTION

Similar Papers

ToxiLab: How Well Do Open-Source LLMs Generate Synthetic Toxicity Data?

83% match2024

Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directions

82% match2025

Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages

81% match1 repo12025

Efficient Toxicity Detection in Gaming Chats: A Comparative Study of Embeddings, Fine-Tuned Transformers and LLMs

81% match2025

Moderating Harm: Benchmarking Large Language Models for Cyberbullying Detection in YouTube Comments

81% match2025

Efficient Detection of Toxic Prompts in Large Language Models

81% match2024

Cite this paper

@article{assoudi2025comparative,
  title  = {A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs (OpenAI, Mistral, Anthropic)},
  author = {Hicham Assoudi},
  year   = {2025},
  eprint = {2505.04640},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.04640v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.04640.svg)](https://codeofpaper.com/paper/2505.04640)

Discussion