Trained MT Metrics Learn to Cope with Machine-translated References

Jannis Vamvas, Tobias Domhan, Sony Trenous, Rico Sennrich, Eva Hasler
12/1/2023
cs.CL

No official code found. Best implementation: amazon-science/prism-finetuned (0 stars, last commit 2024-06-18).

Abstract

Neural metrics trained on human evaluations of MT tend to correlate well with human judgments, but their behavior is not fully understood. In this paper, we perform a controlled experiment and compare a baseline metric that has not been trained on human evaluations (Prism) to a trained version of the same metric (Prism+FT). Surprisingly, we find that Prism+FT becomes more robust to machine-translated references, which are a notorious problem in MT evaluation. This suggests that the effects of metric training go beyond the intended effect of improving overall correlation with human judgments.

View on arXivView PDF

Code Implementations(1)

amazon-science/prism-finetunedCommunity100%
01Shell, PythonNov 15, 20232 years agoNOASSERTION

Similar Papers

Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

88% match1 repo42023

BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training

86% match1 repo42023

Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress

85% match1 repo32025

BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation

85% match1 repo212023

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics

84% match2024

NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist

84% match1 repo72023

Cite this paper

@article{vamvas2023trained,
  title  = {Trained MT Metrics Learn to Cope with Machine-translated References},
  author = {Jannis Vamvas and Tobias Domhan and Sony Trenous and Rico Sennrich and Eva Hasler},
  year   = {2023},
  eprint = {2312.00536},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.00536v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.00536.svg)](https://codeofpaper.com/paper/2312.00536)

Discussion