A Call for Standardization and Validation of Text Style Transfer Evaluation

Phil Ostheimer, Mayank Nagda, Marius Kloft, Sophie Fellenz
6/1/2023
cs.LGcs.CL

No confident code implementation found for this paper.

Abstract

Text Style Transfer (TST) evaluation is, in practice, inconsistent. Therefore, we conduct a meta-analysis on human and automated TST evaluation and experimentation that thoroughly examines existing literature in the field. The meta-analysis reveals a substantial standardization gap in human and automated evaluation. In addition, we also find a validation gap: only few automated metrics have been validated using human experiments. To this end, we thoroughly scrutinize both the standardization and validation gap and reveal the resulting pitfalls. This work also paves the way to close the standardization and validation gap in TST evaluation by calling out requirements to be met by future research.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

APPLS: Evaluating Evaluation Metrics for Plain Language Summarization

84% match1 repo32023

PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

83% match1 repo2026

Hierarchical Evaluation Framework: Best Practices for Human Evaluation

83% match2023

Correction of Errors in Preference Ratings from Automated Metrics for Text Generation

82% match2023

Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics

82% match1 repo122026

A Comprehensive Survey of Action Quality Assessment: Method and Benchmark

82% match2024

Cite this paper

@article{ostheimer2023call,
  title  = {A Call for Standardization and Validation of Text Style Transfer Evaluation},
  author = {Phil Ostheimer and Mayank Nagda and Marius Kloft and Sophie Fellenz},
  year   = {2023},
  eprint = {2306.00539},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.00539v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.00539.svg)](https://codeofpaper.com/paper/2306.00539)

Discussion