MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation

Javier García Gilabert, Carlos Escolano, Audrey Mash, Xixian Liao, Maite Melero
12/16/2024
cs.CL

No official code found. Best implementation: langtech-bsc/mt-evaluation (12 stars, MIT, last commit 2026-04-21).

Abstract

We introduce MT-LENS, a framework designed to evaluate Machine Translation (MT) systems across a variety of tasks, including translation quality, gender bias detection, added toxicity, and robustness to misspellings. While several toolkits have become very popular for benchmarking the capabilities of Large Language Models (LLMs), existing evaluation tools often lack the ability to thoroughly assess the diverse aspects of MT performance. MT-LENS addresses these limitations by extending the capabilities of LM-eval-harness for MT, supporting state-of-the-art datasets and a wide range of evaluation metrics. It also offers a user-friendly platform to compare systems and analyze translations with interactive visualizations. MT-LENS aims to broaden access to evaluation strategies that go beyond traditional translation quality evaluation, enabling researchers and engineers to better understand the performance of a NMT model and also easily measure system's biases.

View on arXivView PDF

Code Implementations(1)

langtech-bsc/mt-evaluationCommunity100%
MIT

A framework for evaluating Machine Translation models.

124C++, ShellOct 7, 20246 months agoMIT

Similar Papers

What do Large Language Models Need for Machine Translation Evaluation?

87% match1 repo42024

Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation

87% match2025

CATER: Leveraging LLM to Pioneer a Multidimensional, Reference-Independent Paradigm in Translation Quality Evaluation

85% match2024

Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling

85% match1 repo162025

The power of Prompts: Evaluating and Mitigating Gender Bias in MT with LLMs

85% match2024

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics

84% match2024

Cite this paper

@article{gilabert2024mtlens,
  title  = {MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation},
  author = {Javier García Gilabert and Carlos Escolano and Audrey Mash and Xixian Liao and Maite Melero},
  year   = {2024},
  eprint = {2412.11615},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.11615v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.11615.svg)](https://codeofpaper.com/paper/2412.11615)

Discussion