We Should Evaluate Real-World Impact

Ehud Reiter
7/8/2025
cs.CL

No confident code implementation found for this paper. 3 possible matches can be shown below.

Abstract

The ACL community has very little interest in evaluating the real-world impact of NLP systems. A structured survey of the ACL Anthology shows that perhaps 0.1% of its papers contain such evaluations; furthermore most papers which include impact evaluations present them very sketchily and instead focus on metric evaluations. NLP technology would be more useful and more quickly adopted if we seriously tried to understand and evaluate its real-world impact.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 3 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Internal and External Impacts of Natural Language Processing Papers

81% match2025

From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP

76% match1 repo72024

The Validity of Coreference-based Evaluations of Natural Language Understanding

76% match2026

Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

75% match1 repo22024

NLP for Social Good: A Survey and Outlook of Challenges, Opportunities, and Responsible Deployment

74% match2025

A Causal Inference Approach for Quantifying Research Impact

74% match2025

Cite this paper

@article{reiter2025should,
  title  = {We Should Evaluate Real-World Impact},
  author = {Ehud Reiter},
  year   = {2025},
  eprint = {2507.05973},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.05973v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.05973.svg)](https://codeofpaper.com/paper/2507.05973)

Discussion