A Measure of the System Dependence of Automated Metrics

Pius von Däniken, Jan Deriu, Mark Cieliebak
12/4/2024
cs.CLcs.AI

No confident code implementation found for this paper. 4 possible matches can be shown below.

Abstract

Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 4 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Evaluating Automatic Metrics with Incremental Machine Translation Systems

86% match1 repo12024

Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress

85% match1 repo32025

Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation

85% match2024

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics

85% match2024

Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!

83% match1 repo102024

Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation

83% match2024

Cite this paper

@article{dniken2024measure,
  title  = {A Measure of the System Dependence of Automated Metrics},
  author = {Pius von Däniken and Jan Deriu and Mark Cieliebak},
  year   = {2024},
  eprint = {2412.03152},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.03152v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.03152.svg)](https://codeofpaper.com/paper/2412.03152)

Discussion