MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibration

David Anugraha, Garry Kuwanto, Lucky Susanto, Derry Tanti Wijaya, Genta Indra Winata
11/1/2024
cs.CLcs.AIcs.LG

No official code found. Best implementation: meta-metrics/metametrics (15 stars, Apache-2.0, last commit 2024-12-30).

Abstract

We present MetaMetrics-MT, an innovative metric designed to evaluate machine translation (MT) tasks by aligning closely with human preferences through Bayesian optimization with Gaussian Processes. MetaMetrics-MT enhances existing MT metrics by optimizing their correlation with human judgments. Our experiments on the WMT24 metric shared task dataset demonstrate that MetaMetrics-MT outperforms all existing baselines, setting a new benchmark for state-of-the-art performance in the reference-based setting. Furthermore, it achieves comparable results to leading metrics in the reference-free setting, offering greater efficiency.

View on arXivView PDF

Code Implementations(1)

meta-metrics/metametricsCommunity100%
Apache-2.0

Accepted to ICLR 2025. MetaMetrics is a calibrated meta-metric designed to evaluate generation tasks across different modalities aligned with alignment with human preferences.

151PythonSep 23, 20241 years agoApache-2.0

Similar Papers

Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!

88% match1 repo102024

MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences

87% match1 repo152024

MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences

87% match1 repo152026

Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling

87% match1 repo162025

Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation

86% match2024

Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translation

86% match2025

Cite this paper

@article{anugraha2024metametricsmt,
  title  = {MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibration},
  author = {David Anugraha and Garry Kuwanto and Lucky Susanto and Derry Tanti Wijaya and Genta Indra Winata},
  year   = {2024},
  eprint = {2411.00390},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.00390v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.00390.svg)](https://codeofpaper.com/paper/2411.00390)

Discussion