The 2nd FutureDial Challenge: Dialog Systems with Retrieval Augmented Generation (FutureDial-RAG)

Yucheng Cai, Si Chen, Yuxuan Wu, Yi Huang, Junlan Feng +1 more
5/21/2024
cs.CLcs.AI

No official code found. Best implementation: futuredialchallenge/2024-RAG (11 stars, last commit 2024-08-10).

Abstract

Recently, increasing research interests have focused on retrieval augmented generation (RAG) to mitigate hallucination for large language models (LLMs). Following this trend, we launch the FutureDial-RAG challenge at SLT 2024, which aims at promoting the study of RAG for dialog systems. The challenge builds upon the MobileCS2 dataset, a real-life customer service datasets with nearly 3000 high-quality dialogs containing annotations for knowledge base query and corresponding results. Over the dataset, we define two tasks, track 1 for knowledge retrieval and track 2 for response generation, which are core research questions in dialog systems with RAG. We build baseline systems for the two tracks and design metrics to measure whether the systems can perform accurate retrieval and generate informative and coherent response. The baseline results show that it is very challenging to perform well on the two tasks, which encourages the participating teams and the community to study how to make better use of RAG for real-life dialog systems.

View on arXivView PDF

Code Implementations(1)

futuredialchallenge/2024-RAGCommunity100%

A Challenge on Dialog Systems with Retrieval Augmented Generation (FutureDial-RAG), Co-located with SLT2024 FutureDial-RAG Challenge

113Shell, PythonApr 30, 20242 years ago

Similar Papers

MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

87% match1 repo1452025

RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation

86% match1 repo3142024

A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions

85% match2024

CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation

85% match1 repo682024

RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues

85% match1 repo82024

A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models

85% match2024

Cite this paper

@article{cai2024futuredial,
  title  = {The 2nd FutureDial Challenge: Dialog Systems with Retrieval Augmented Generation (FutureDial-RAG)},
  author = {Yucheng Cai and Si Chen and Yuxuan Wu and Yi Huang and Junlan Feng and Zhijian Ou},
  year   = {2024},
  eprint = {2405.13084},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.13084v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.13084.svg)](https://codeofpaper.com/paper/2405.13084)

Discussion