Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

Patrícia Schmidtová, Saad Mahamood, Simone Balloccu, Ondřej Dušek, Albert Gatt +4 more
8/17/2024
cs.CL

No official code found. Best implementation: patuchen/nlg_metric_usage (2 stars, last commit 2024-09-26).

Abstract

Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use of automatic metrics, focusing particularly on natural language generation (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, including inappropriate metric usage, lack of implementation details and missing correlations with human judgements. We conclude with recommendations that we believe authors should follow to enable more rigour within the field.

View on arXivView PDF

Code Implementations(1)

patuchen/nlg_metric_usageCommunity100%

Examining the usage of metrics for NLG in 2023 papers

21Jupyter NotebookMay 14, 20242 years ago

Similar Papers

Reference-free Evaluation Metrics for Text Generation: A Survey

90% match2025

The illusion of a perfect metric: Why evaluating AI's words is harder than it looks

86% match2025

LUNA: A Framework for Language Understanding and Naturalness Assessment

83% match1 repo122024

LLM-based NLG Evaluation: Current Status and Challenges

83% match2024

A Measure of the System Dependence of Automated Metrics

82% match2024

Evaluation Metrics for Text Data Augmentation in NLP

82% match2024

Cite this paper

@article{schmidtov2024automatic,
  title  = {Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices},
  author = {Patrícia Schmidtová and Saad Mahamood and Simone Balloccu and Ondřej Dušek and Albert Gatt and Dimitra Gkatzia and David M. Howcroft and Ondřej Plátek and Adarsa Sivaprasad},
  year   = {2024},
  eprint = {2408.09169},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.09169v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.09169.svg)](https://codeofpaper.com/paper/2408.09169)

Discussion