Interactive Benchmarks

Baoqing Yue, Zihan Zhu, Yifan Zhang, Jichen Feng, Hufei Yang +1 more
3/5/2026
cs.AIcs.CLcs.LG

Official code: interactivebench/InteractiveBench (31 stars, MIT, last commit 2026-03-29).

Abstract

Standard benchmarks have become increasingly unreliable due to saturation, subjectivity, and poor generalization. We argue that evaluating model's ability to acquire information actively is important to assess model's intelligence. We propose Interactive Benchmarks, a unified evaluation paradigm that assesses model's reasoning ability in an interactive process under budget constraints. We instantiate this framework across two settings: Interactive Proofs, where models interact with a judge to deduce objective truths or answers in logic and mathematics; and Interactive Games, where models reason strategically to maximize long-horizon utilities. Our results show that interactive benchmarks provide a robust and faithful assessment of model intelligence, revealing that there is still substantial room to improve in interactive scenarios. Project page: https://github.com/interactivebench/interactivebench

View on arXivView PDF

Code Implementations(1)

interactivebench/InteractiveBenchOfficial100%
MIT

Official Project Page for Interactive Benchmarks

310CSS, HTMLMar 2, 20266 months agoMIT

Similar Papers

Benchmark-Driven Selection of AI: Evidence from DeepSeek-R1

84% match2025

Watson & Holmes: A Naturalistic Benchmark for Comparing Human and LLM Reasoning

84% match2026

On the Reasoning Capacity of AI Models and How to Quantify It

84% match2025

Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap

82% match1 repo902024

What Does Your Benchmark Really Measure? A Framework for Robust Inference of AI Capabilities

82% match2025

Benchmarking at the Edge of Comprehension

82% match2026

Cite this paper

@article{yue2026interactive,
  title  = {Interactive Benchmarks},
  author = {Baoqing Yue and Zihan Zhu and Yifan Zhang and Jichen Feng and Hufei Yang and Mengdi Wang},
  year   = {2026},
  eprint = {2603.04737},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2603.04737v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2603.04737.svg)](https://codeofpaper.com/paper/2603.04737)

Discussion