Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation

Malik Marmonier, Benoît Sagot, Rachel Bawden
3/4/2026
cs.CL

No official code found. Best implementation: mmarmonier/Hindex (0 stars, MIT, last commit 2026-03-05).

Abstract

This paper investigates two complementary paradigms for predicting machine translation (MT) quality: source-side difficulty prediction and candidate-side quality estimation (QE). The rapid adoption of Large Language Models (LLMs) into MT workflows is reshaping the research landscape, yet its impact on established quality prediction paradigms remains underexplored. We study this issue through a series of "hindsight" experiments on a unique, multi-candidate dataset resulting from a genuine MT post-editing (MTPE) project. The dataset consists of over 6,000 English source segments with nine translation hypotheses from a diverse set of traditional neural MT systems and advanced LLMs, all evaluated against a single, final human post-edited reference. Using Kendall's rank correlation, we assess the predictive power of source-side difficulty metrics, candidate-side QE models and position heuristics against two gold-standard scores: TER (as a proxy for post-editing effort) and COMET (as a proxy for human judgment). Our findings highlight that the architectural shift towards LLMs alters the reliability of established quality prediction methods while simultaneously mitigating previous challenges in document-level translation.

View on arXivView PDF

Code Implementations(1)

mmarmonier/HindexCommunity66%
MIT

Code and data for "Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation." Includes analyses of source-side difficulty prediction, reference-free QE, and positional bias in doc2doc LLM vs NMT translation systems.

00Jupyter NotebookMar 2, 20267 months agoMIT

Similar Papers

Introducing Quality Estimation to Machine Translation Post-editing Workflow: An Empirical Study on Its Usefulness

88% match1 repo2025

QE4PE: Word-level Quality Estimation for Human Post-Editing

88% match1 repo52025

Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation

86% match1 repo82024

Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations

85% match1 repo142024

CATER: Leveraging LLM to Pioneer a Multidimensional, Reference-Independent Paradigm in Translation Quality Evaluation

85% match2024

Are Large Language Models State-of-the-art Quality Estimators for Machine Translation of User-generated Content?

85% match1 repo2024

Cite this paper

@article{marmonier2026hindsight,
  title  = {Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation},
  author = {Malik Marmonier and Benoît Sagot and Rachel Bawden},
  year   = {2026},
  eprint = {2603.04083},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2603.04083v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2603.04083.svg)](https://codeofpaper.com/paper/2603.04083)

Discussion