Preliminary WMT24 Ranking of General MT Systems and LLMs

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich +16 more
7/29/2024
cs.CL

No official code found. Best implementation: wmt-conference/wmt-collect-translations (14 stars, Apache-2.0, last commit 2025-09-16).

Abstract

This is the preliminary ranking of WMT24 General MT systems based on automatic metrics. The official ranking will be a human evaluation, which is superior to the automatic ranking and supersedes it. The purpose of this report is not to interpret any findings but only provide preliminary results to the participants of the General MT task that may be useful during the writing of the system submission.

View on arXivView PDF

Code Implementations(1)

wmt-conference/wmt-collect-translationsCommunity100%
Apache-2.0

Toolkit used to collect translations from various online providers and LLMs

143Shell, PythonJul 12, 20241 years agoApache-2.0

Similar Papers

Preliminary Ranking of WMT25 General Machine Translation Systems

87% match2025

Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference

76% match2024

MT-Ranker: Reference-free machine translation evaluation by inter-system ranking

73% match1 repo102026

Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach

73% match2024

Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat

73% match2024

GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking

73% match2024

Cite this paper

@article{kocmi2024preliminary,
  title  = {Preliminary WMT24 Ranking of General MT Systems and LLMs},
  author = {Tom Kocmi and Eleftherios Avramidis and Rachel Bawden and Ondrej Bojar and Anton Dvorkovich and Christian Federmann and Mark Fishel and Markus Freitag and Thamme Gowda and Roman Grundkiewicz and Barry Haddow and Marzena Karpinska and Philipp Koehn and Benjamin Marie and Kenton Murray and Masaaki Nagata and Martin Popel and Maja Popovic and Mariya Shmatova and Steinþór Steingrímsson and Vilém Zouhar},
  year   = {2024},
  eprint = {2407.19884},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.19884v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.19884.svg)](https://codeofpaper.com/paper/2407.19884)

Discussion