Evaluating Automatic Metrics with Incremental Machine Translation Systems

Guojun Wu, Shay B. Cohen, Rico Sennrich
7/3/2024
cs.CL

No official code found. Best implementation: guojun-wu/Evo (1 star, last commit 2024-09-26).

Abstract

We introduce a dataset comprising commercial machine translations, gathered weekly over six years across 12 translation directions. Since human A/B testing is commonly used, we assume commercial systems improve over time, which enables us to evaluate machine translation (MT) metrics based on their preference for more recent translations. Our study not only confirms several prior findings, such as the advantage of neural metrics over non-neural ones, but also explores the debated issue of how MT quality affects metric reliability--an investigation that smaller datasets in previous research could not sufficiently explore. Overall, our research demonstrates the dataset's value as a testbed for metric evaluation. We release our code at https://github.com/gjwubyron/Evo

View on arXivView PDF

Code Implementations(1)

guojun-wu/EvoCommunity100%
10PythonOct 31, 20232 years ago

Similar Papers

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics

89% match2024

Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress

88% match1 repo32025

Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains

87% match1 repo62024

Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!

87% match1 repo102024

Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling

87% match1 repo162025

Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation

86% match2024

Cite this paper

@article{wu2024evaluating,
  title  = {Evaluating Automatic Metrics with Incremental Machine Translation Systems},
  author = {Guojun Wu and Shay B. Cohen and Rico Sennrich},
  year   = {2024},
  eprint = {2407.03277},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.03277v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.03277.svg)](https://codeofpaper.com/paper/2407.03277)

Discussion