Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations

Tiziano Labruna, Sofia Brenna, Andrea Zaninello, Bernardo Magnini
5/23/2023
cs.CLcs.AI

No confident code implementation found for this paper.

Abstract

Large pre-trained language models have exhibited unprecedented capabilities in producing high-quality text via prompting techniques. This fact introduces new possibilities for data collection and annotation, particularly in situations where such data is scarce, complex to gather, expensive, or even sensitive. In this paper, we explore the potential of these models to generate and annotate goal-oriented dialogues, and conduct an in-depth analysis to evaluate their quality. Our experiments employ ChatGPT, and encompass three categories of goal-oriented dialogues (task-oriented, collaborative, and explanatory), two generation modes (interactive and one-shot), and two languages (English and Italian). Based on extensive human-based evaluations, we demonstrate that the quality of generated dialogues and annotations is on par with those generated by humans.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues

85% match1 repo1632023

User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue

85% match2023

Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation

85% match2023

EXMODD: An EXplanatory Multimodal Open-Domain Dialogue dataset

85% match2023

Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System

85% match2023

PLACES: Prompting Language Models for Social Conversation Synthesis

85% match1 repo112023

Cite this paper

@article{labruna2023unraveling,
  title  = {Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations},
  author = {Tiziano Labruna and Sofia Brenna and Andrea Zaninello and Bernardo Magnini},
  year   = {2023},
  eprint = {2305.14556},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.14556v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.14556.svg)](https://codeofpaper.com/paper/2305.14556)

Discussion