Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation

John Mendonça, Patrícia Pereira, Helena Moniz, João Paulo Carvalho, Alon Lavie +1 more
8/31/2023
cs.CL

No official code found. Best implementation: johndmendonca/DialEvalML (4 stars, MIT, last commit 2025-05-16).

Abstract

Despite significant research effort in the development of automatic dialogue evaluation metrics, little thought is given to evaluating dialogues other than in English. At the same time, ensuring metrics are invariant to semantically similar responses is also an overlooked topic. In order to achieve the desired properties of robustness and multilinguality for dialogue evaluation metrics, we propose a novel framework that takes advantage of the strengths of current evaluation models with the newly-established paradigm of prompting Large Language Models (LLMs). Empirical results show our framework achieves state of the art results in terms of mean Spearman correlation scores across several benchmarks and ranks first place on both the Robust and Multilingual tasks of the DSTC11 Track 4 "Automatic Evaluation Metrics for Open-Domain Dialogue Systems", proving the evaluation capabilities of prompted LLMs.

View on arXivView PDF

Code Implementations(1)

johndmendonca/DialEvalMLCommunity100%
MIT
40Shell, PythonJul 12, 20231 years agoMIT

Similar Papers

A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators

86% match1 repo52023

xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark

85% match1 repo72023

Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information

84% match1 repo42023

Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4

83% match1 repo92023

Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations

83% match2023

SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation

83% match1 repo32024

Cite this paper

@article{mendona2023simple,
  title  = {Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation},
  author = {John Mendonça and Patrícia Pereira and Helena Moniz and João Paulo Carvalho and Alon Lavie and Isabel Trancoso},
  year   = {2023},
  eprint = {2308.16797},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.16797v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.16797.svg)](https://codeofpaper.com/paper/2308.16797)

Discussion