G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu +1 more
3/29/2023
cs.CLcs.AI

Official code: nlpyang/geval (430 stars, MIT, last commit 2024-02-04). 1 more implementation below.

Abstract

The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs. We experiment with two generation tasks, text summarization and dialogue generation. We show that G-Eval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin. We also propose preliminary analysis on the behavior of LLM-based evaluators, and highlight the potential issue of LLM-based evaluators having a bias towards the LLM-generated texts. The code is at https://github.com/nlpyang/geval

View on arXivView PDF

Code Implementations(2)

nlpyang/gevalOfficial100%
MIT

Code for paper "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment"

43046PythonMay 23, 20232 years agoMIT
K010TE/replicacao-gevalCommunity59%
MIT

Este repositório contém a **replicação do experimento descrito no artigo [G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment](https://arxiv.org/abs/2303.16634)**, com o objetivo de avaliar a capacidade de modelos de linguagem em julgar automaticamente a qualidade de saídas de sistemas de Geração de Linguagem Natural

00Python, Jupyter NotebookJun 23, 20251 years agoMIT

Similar Papers

LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores

89% match2023

CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation

89% match1 repo1482023

DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering

89% match1 repo62023

Is ChatGPT a Good NLG Evaluator? A Preliminary Study

88% match1 repo432023

Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study

88% match2023

INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback

88% match1 repo622023

Cite this paper

@article{liu2023geval,
  title  = {G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment},
  author = {Yang Liu and Dan Iter and Yichong Xu and Shuohang Wang and Ruochen Xu and Chenguang Zhu},
  year   = {2023},
  eprint = {2303.16634},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.16634v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.16634.svg)](https://codeofpaper.com/paper/2303.16634)

Discussion