MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa +1 more
2/26/2026
cs.CL

Official code: IBM/mt-rag-benchmark (145 stars, Apache-2.0, last commit 2026-04-30).

Abstract

We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models. We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora. Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. Our benchmark is available at https://github.com/IBM/mt-rag-benchmark

View on arXivView PDF

Code Implementations(1)

IBM/mt-rag-benchmarkOfficial100%
Apache-2.0

Multi-Turn RAG Benchmark

14529PythonDec 17, 20245 months agoApache-2.0

Similar Papers

MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

94% match1 repo1452025

CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation

88% match1 repo682024

CRAG - Comprehensive RAG Benchmark

85% match1 repo3112026

RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues

84% match1 repo82024

HANRAG: Heuristic Accurate Noise-resistant Retrieval-Augmented Generation for Multi-hop Question Answering

84% match2025

MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

84% match1 repo4682024

Cite this paper

@article{rosenthal2026mtragun,
  title  = {MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations},
  author = {Sara Rosenthal and Yannis Katsis and Vraj Shah and Lihong He and Lucian Popa and Marina Danilevsky},
  year   = {2026},
  eprint = {2602.23184},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2602.23184v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2602.23184.svg)](https://codeofpaper.com/paper/2602.23184)

Discussion