Online vs Offline: A Comparative Study of First-Party and Third-Party Evaluations of Social Chatbots

Ekaterina Svikhnushina, Pearl Pu
9/12/2024
cs.HCcs.CL

No confident code implementation found for this paper.

Abstract

This paper explores the efficacy of online versus offline evaluation methods in assessing conversational chatbots, specifically comparing first-party direct interactions with third-party observational assessments. By extending a benchmarking dataset of user dialogs with empathetic chatbots with offline third-party evaluations, we present a systematic comparison between the feedback from online interactions and the more detached offline third-party evaluations. Our results reveal that offline human evaluations fail to capture the subtleties of human-chatbot interactions as effectively as online assessments. In comparison, automated third-party evaluations using a GPT-4 model offer a better approximation of first-party human judgments given detailed instructions. This study highlights the limitations of third-party evaluations in grasping the complexities of user experiences and advocates for the integration of direct interaction feedback in conversational AI evaluation to enhance system development and user satisfaction.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Dialogue You Can Trust: Human and AI Perspectives on Generated Conversations

84% match2024

Designing and Evaluating Multi-Chatbot Interface for Human-AI Communication: Preliminary Findings from a Persuasion Task

82% match2024

The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches

82% match2024

Rewarding Chatbots for Real-World Engagement with Millions of Users

81% match2023

Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs

80% match1 repo2024

First-Person Fairness in Chatbots

80% match2024

Cite this paper

@article{svikhnushina2024online,
  title  = {Online vs Offline: A Comparative Study of First-Party and Third-Party Evaluations of Social Chatbots},
  author = {Ekaterina Svikhnushina and Pearl Pu},
  year   = {2024},
  eprint = {2409.07823},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.07823v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.07823.svg)](https://codeofpaper.com/paper/2409.07823)

Discussion