A Two-dimensional Zero-shot Dialogue State Tracking Evaluation Method using GPT-4

Ming Gu, Yan Yang
6/17/2024
cs.CL

No official code found. Best implementation: SLEEPWALKERG/LLM-DST-EVAL (0 stars, last commit 2024-05-25).

Abstract

Dialogue state tracking (DST) is evaluated by exact matching methods, which rely on large amounts of labeled data and ignore semantic consistency, leading to over-evaluation. Currently, leveraging large language models (LLM) in evaluating natural language processing tasks has achieved promising results. However, using LLM for DST evaluation is still under explored. In this paper, we propose a two-dimensional zero-shot evaluation method for DST using GPT-4, which divides the evaluation into two dimensions: accuracy and completeness. Furthermore, we also design two manual reasoning paths in prompting to further improve the accuracy of evaluation. Experimental results show that our method achieves better performance compared to the baselines, and is consistent with traditional exact matching based methods.

View on arXivView PDF

Code Implementations(1)

SLEEPWALKERG/LLM-DST-EVALCommunity100%

This is the code for our proposed LLM-based two-dimensional evaluation method for dialogue state tracking.

00PythonMay 25, 20242 years ago

Similar Papers

A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding

86% match1 repo32024

Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMs

86% match2025

Beyond Single-User Dialogue: Assessing Multi-User Dialogue State Tracking Capabilities of Large Language Models

85% match2025

Large Language Models as Zero-shot Dialogue State Tracker through Function Calling

83% match1 repo712024

Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking

83% match1 repo42023

Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation

82% match1 repo152024

Cite this paper

@article{gu2024twodimensional,
  title  = {A Two-dimensional Zero-shot Dialogue State Tracking Evaluation Method using GPT-4},
  author = {Ming Gu and Yan Yang},
  year   = {2024},
  eprint = {2406.11651},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.11651v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.11651.svg)](https://codeofpaper.com/paper/2406.11651)

Discussion