Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval

Yohan Lee, Yongwoo Song, Sangyeop Kim
10/3/2025
cs.CL

Official code: l-yohai/CDR-Benchmark (3 stars, last commit 2025-09-30).

Abstract

We present the Conversational Data Retrieval (CDR) benchmark, the first comprehensive test set for evaluating systems that retrieve conversation data for product insights. With 1.6k queries across five analytical tasks and 9.1k conversations, our benchmark provides a reliable standard for measuring conversational data retrieval performance. Our evaluation of 16 popular embedding models shows that even the best models reach only around NDCG@10 of 0.51, revealing a substantial gap between document and conversational data retrieval capabilities. Our work identifies unique challenges in conversational data retrieval (implicit state recognition, turn dynamics, contextual references) while providing practical query templates and detailed error analysis across different task categories. The benchmark dataset and code are available at https://github.com/l-yohai/CDR-Benchmark.

View on arXivView PDF

Code Implementations(1)

l-yohai/CDR-BenchmarkOfficial100%

Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval (EMNLP 2025 Industry Track)

31Shell, PythonSep 30, 20251 years agoNOASSERTION

Similar Papers

Convomem Benchmark: Why Your First 150 Conversations Don't Need RAG

83% match2025

MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks

83% match1 repo2026

CONVERSER: Few-Shot Conversational Dense Retrieval with Synthetic Data Generation

83% match1 repo82023

CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation

82% match1 repo682024

Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries

81% match2024

A Survey on Recent Advances in Conversational Data Generation

81% match2024

Cite this paper

@article{lee2025finding,
  title  = {Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval},
  author = {Yohan Lee and Yongwoo Song and Sangyeop Kim},
  year   = {2025},
  eprint = {2510.02938},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.02938v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.02938.svg)](https://codeofpaper.com/paper/2510.02938)

Discussion