HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models

Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yulong Chen +1 more
10/25/2023
cs.CLcs.AI

No official code found. Best implementation: ying-hui-he/Hi-ToM_dataset (21 stars, Apache-2.0, last commit 2025-10-11).

Abstract

Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. ToM plays a critical role in the development of intelligence, language understanding, and cognitive processes. While previous work has primarily focused on first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others' beliefs. We introduce HI-TOM, a Higher Order Theory of Mind benchmark. Our experimental evaluation using various Large Language Models (LLMs) indicates a decline in performance on higher-order ToM tasks, demonstrating the limitations of current LLMs. We conduct a thorough analysis of different failure cases of LLMs, and share our thoughts on the implications of our findings on the future of NLP.

View on arXivView PDF

Code Implementations(1)

ying-hui-he/Hi-ToM_datasetCommunity100%
Apache-2.0
211Shell, PythonJun 5, 20231 years agoApache-2.0

Similar Papers

How FaR Are Large Language Models From Agents with Theory-of-Mind?

88% match2023

Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs

86% match2023

MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic

86% match1 repo212023

Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Tests

86% match2023

Boosting Theory-of-Mind Performance in Large Language Models via Prompting

86% match1 repo312023

The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inference

85% match1 repo12025

Cite this paper

@article{he2023hitom,
  title  = {HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models},
  author = {Yinghui He and Yufan Wu and Yilin Jia and Rada Mihalcea and Yulong Chen and Naihao Deng},
  year   = {2023},
  eprint = {2310.16755},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.16755v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.16755.svg)](https://codeofpaper.com/paper/2310.16755)

Discussion