Multidimensional Evaluation for Text Style Transfer Using ChatGPT

Huiyuan Lai, Antonio Toral, Malvina Nissim
4/26/2023
cs.CL

No official code found. Best implementation: laihuiyuan/eval-formality-transfer (5 stars, MIT, last commit 2023-04-27).

Abstract

We investigate the potential of ChatGPT as a multidimensional evaluator for the task of \emph{Text Style Transfer}, alongside, and in comparison to, existing automatic metrics as well as human judgements. We focus on a zero-shot setting, i.e. prompting ChatGPT with specific task instructions, and test its performance on three commonly-used dimensions of text style transfer evaluation: style strength, content preservation, and fluency. We perform a comprehensive correlation analysis for two transfer directions (and overall) at different levels. Compared to existing automatic metrics, ChatGPT achieves competitive correlations with human judgments. These preliminary results are expected to provide a first glimpse into the role of large language models in the multidimensional evaluation of stylized text generation.

View on arXivView PDF

Code Implementations(1)

laihuiyuan/eval-formality-transferCommunity100%
MIT

Multidimensional Evaluation for Text Style Transfer Using ChatGPT. Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer (HumEval 2022)

53Perl, PythonMar 14, 20223 years agoMIT
automatic-evaluationformality-style-transferhuman-evaluationtext-style-transfer

Similar Papers

Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training

86% match1 repo42023

Text Style Transfer Evaluation Using Large Language Models

86% match2023

Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification

83% match2025

Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?

83% match2025

Multi-Dimensional Evaluation of Text Summarization with In-Context Learning

83% match1 repo42023

A Closer Look into Automatic Evaluation Using Large Language Models

83% match1 repo192023

Cite this paper

@article{lai2023multidimensional,
  title  = {Multidimensional Evaluation for Text Style Transfer Using ChatGPT},
  author = {Huiyuan Lai and Antonio Toral and Malvina Nissim},
  year   = {2023},
  eprint = {2304.13462},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.13462v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.13462.svg)](https://codeofpaper.com/paper/2304.13462)

Discussion