GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models

Dylan Hutson, Daniel Vennemeyer, Aneesh Deshmukh, Justin Zhan, Tianyu Jiang
9/23/2025
cs.CLcs.AI

No official code found. Best implementation: cincynlp/GuessingGame (1 star, last commit 2025-10-06).

Abstract

We introduce GuessingGame, a protocol for evaluating large language models (LLMs) as strategic question-askers in open-ended, open-domain settings. A Guesser LLM identifies a hidden object by posing free-form questions to an Oracle without predefined choices or candidate lists. To measure question quality, we propose two information gain (IG) metrics: a Bayesian method that tracks belief updates over semantic concepts using LLM-scored relevance, and an entropy-based method that filters candidates via ConceptNet. Both metrics are model-agnostic and support post hoc analysis. Across 858 games with multiple models and prompting strategies, higher IG strongly predicts efficiency: a one-standard-deviation IG increase reduces expected game length by 43\%. Prompting constraints guided by IG, such as enforcing question diversity, enable weaker models to significantly improve performance. These results show that question-asking in LLMs is both measurable and improvable, and crucial for interactive reasoning.

View on arXivView PDF

Code Implementations(1)

cincynlp/GuessingGameCommunity68%

Code for the paper: GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models.

10Shell, PythonSep 19, 20251 years ago

Similar Papers

Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain

88% match1 repo32024

Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling

86% match2024

Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games

85% match1 repo392023

What Would You Ask When You First Saw $a^2+b^2=c^2$? Evaluating LLM on Curiosity-Driven Questioning

85% match2024

Do Large Language Models have Problem-Solving Capability under Incomplete Information Scenarios?

84% match2024

BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design

84% match2025

Cite this paper

@article{hutson2025guessinggame,
  title  = {GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models},
  author = {Dylan Hutson and Daniel Vennemeyer and Aneesh Deshmukh and Justin Zhan and Tianyu Jiang},
  year   = {2025},
  eprint = {2509.19593},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.19593v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.19593.svg)](https://codeofpaper.com/paper/2509.19593)

Discussion