Reliability Crisis of Reference-free Metrics for Grammatical Error Correction

Takumi Goto, Yusuke Sakai, Taro Watanabe
9/30/2025
cs.CL

No official code found. Best implementation: gotutiyan/attack-gec-metrics (0 stars, MIT, last commit 2025-10-01).

Abstract

Reference-free evaluation metrics for grammatical error correction (GEC) have achieved high correlation with human judgments. However, these metrics are not designed to evaluate adversarial systems that aim to obtain unjustifiably high scores. The existence of such systems undermines the reliability of automatic evaluation, as it can mislead users in selecting appropriate GEC systems. In this study, we propose adversarial attack strategies for four reference-free metrics: SOME, Scribendi, IMPARA, and LLM-based metrics, and demonstrate that our adversarial systems outperform the current state-of-the-art. These findings highlight the need for more robust evaluation methods.

View on arXivView PDF

Code Implementations(1)

gotutiyan/attack-gec-metricsCommunity63%
MIT

Code for the paper: "Reliability Crisis of Reference-free Metrics for Grammatical Error Correction", EMNLP 2025 Findings.

00PythonSep 3, 20251 years agoMIT

Similar Papers

Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?

86% match1 repo142025

CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction

85% match1 repo122024

Measuring the Robustness of Reference-Free Dialogue Evaluation Systems

84% match1 repo2025

DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models

84% match1 repo62024

Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction

83% match2024

Revisiting Meta-evaluation for Grammatical Error Correction

83% match1 repo32024

Cite this paper

@article{goto2025reliability,
  title  = {Reliability Crisis of Reference-free Metrics for Grammatical Error Correction},
  author = {Takumi Goto and Yusuke Sakai and Taro Watanabe},
  year   = {2025},
  eprint = {2509.25961},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.25961v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.25961.svg)](https://codeofpaper.com/paper/2509.25961)

Discussion