How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation

Huda Khayrallah, Zuhaib Akhtar, Edward Cohen, Jyothir S, João Sedoc
5/23/2023
cs.CL

No confident code implementation found for this paper.

Abstract

We release MMSMR, a Massively Multi-System MultiReference dataset to enable future work on metrics and evaluation for dialog. Automatic metrics for dialogue evaluation should be robust proxies for human judgments; however, the verification of robustness is currently far from satisfactory. To quantify the robustness correlation and understand what is necessary in a test set, we create and release an 8-reference dialog dataset by extending single-reference evaluation sets and introduce this new language learning conversation dataset. We then train 1750 systems and evaluate them on our novel test set and the DailyDialog dataset. We release the novel test set, and model hyper parameters, inference outputs, and metric scores for each system on a variety of datasets.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

84% match2026

xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark

84% match1 repo72023

EXMODD: An EXplanatory Multimodal Open-Domain Dialogue dataset

84% match2023

Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4

83% match1 repo92023

DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI

83% match1 repo5272023

QualEval: Qualitative Evaluation for Model Improvement

83% match2023

Cite this paper

@article{khayrallah2023choose,
  title  = {How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation},
  author = {Huda Khayrallah and Zuhaib Akhtar and Edward Cohen and Jyothir S and João Sedoc},
  year   = {2023},
  eprint = {2305.14533},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.14533v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.14533.svg)](https://codeofpaper.com/paper/2305.14533)

Discussion