CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation

Md Mahfuz Ibn Alam, Sina Ahmadi, Antonios Anastasopoulos
5/26/2023
cs.CL

No confident code implementation found for this paper.

Abstract

Neural machine translation (NMT) systems exhibit limited robustness in handling source-side linguistic variations. Their performance tends to degrade when faced with even slight deviations in language usage, such as different domains or variations introduced by second-language speakers. It is intuitive to extend this observation to encompass dialectal variations as well, but the work allowing the community to evaluate MT systems on this dimension is limited. To alleviate this issue, we compile and release CODET, a contrastive dialectal benchmark encompassing 891 different variations from twelve different languages. We also quantitatively demonstrate the challenges large MT models face in effectively translating dialectal variants. All the data and code have been released.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A Benchmark for Evaluating Machine Translation Metrics on Dialects Without Standard Orthography

87% match2 repos872023

Dialect Transfer for Swiss German Speech Translation

87% match2023

Improving Term Evaluation in Machine Translation: Variation Matters

86% match2026

DialUp! Modeling the Language Continuum by Adapting Models to Dialects and Dialects to Models

86% match2025

Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark

86% match2025

Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction

86% match2023

Cite this paper

@article{alam2023codet,
  title  = {CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation},
  author = {Md Mahfuz Ibn Alam and Sina Ahmadi and Antonios Anastasopoulos},
  year   = {2023},
  eprint = {2305.17267},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.17267v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.17267.svg)](https://codeofpaper.com/paper/2305.17267)

Discussion