How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages?

Anushka Singh, Ananya B. Sai, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan +1 more
6/6/2024
cs.CL

No official code found. Best implementation: AI4Bharat/IndicMT-Eval (6 stars, last commit 2024-09-15).

Abstract

While machine translation evaluation has been studied primarily for high-resource languages, there has been a recent interest in evaluation for low-resource languages due to the increasing availability of data and models. In this paper, we focus on a zero-shot evaluation setting focusing on low-resource Indian languages, namely Assamese, Kannada, Maithili, and Punjabi. We collect sufficient Multi-Dimensional Quality Metrics (MQM) and Direct Assessment (DA) annotations to create test sets and meta-evaluate a plethora of automatic evaluation metrics. We observe that even for learned metrics, which are known to exhibit zero-shot performance, the Kendall Tau and Pearson correlations with human annotations are only as high as 0.32 and 0.45. Synthetic data approaches show mixed results and overall do not help close the gap by much for these languages. This indicates that there is still a long way to go for low-resource evaluation.

View on arXivView PDF

Code Implementations(1)

AI4Bharat/IndicMT-EvalCommunity100%

IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages, ACL 2023

61CSS, HTMLMay 24, 20232 years ago

Similar Papers

Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Languages

86% match2024

Trainable Reference-Based Evaluation Metric for Identifying Quality of English-Gujarati Machine Translation System

85% match2025

LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark

84% match1 repo2025

Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics

83% match2026

Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages

83% match2025

Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings

83% match2024

Cite this paper

@article{singh2024good,
  title  = {How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages?},
  author = {Anushka Singh and Ananya B. Sai and Raj Dabre and Ratish Puduppully and Anoop Kunchukuttan and Mitesh M Khapra},
  year   = {2024},
  eprint = {2406.03893},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.03893v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.03893.svg)](https://codeofpaper.com/paper/2406.03893)

Discussion