Existing LLMs Are Not Self-Consistent For Simple Tasks

Zhenru Lin, Jiawen Tao, Yang Yuan, Andrew Chi-Chih Yao
6/23/2025
cs.CL

Official code: scorpio-nova/llm-self-consistency (9 stars, last commit 2025-10-05).

Abstract

Large Language Models (LLMs) have grown increasingly powerful, yet ensuring their decisions remain transparent and trustworthy requires self-consistency -- no contradictions in their internal reasoning. Our study reveals that even on simple tasks, such as comparing points on a line or a plane, or reasoning in a family tree, all smaller models are highly inconsistent, and even state-of-the-art models like DeepSeek-R1 and GPT-o4-mini are not fully self-consistent. To quantify and mitigate these inconsistencies, we introduce inconsistency metrics and propose two automated methods -- a graph-based and an energy-based approach. While these fixes provide partial improvements, they also highlight the complexity and importance of self-consistency in building more reliable and interpretable AI. The code and data are available at https://github.com/scorpio-nova/llm-self-consistency.

View on arXivView PDF

Code Implementations(1)

scorpio-nova/llm-self-consistencyOfficial100%
91PythonMay 16, 20251 years ago

Similar Papers

ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities

88% match1 repo32025

Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling

87% match2024

Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?

87% match1 repo142024

Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning

87% match2025

When is the consistent prediction likely to be a correct prediction?

86% match2024

Reevaluating Self-Consistency Scaling in Multi-Agent Systems

86% match2025

Cite this paper

@article{lin2025existing,
  title  = {Existing LLMs Are Not Self-Consistent For Simple Tasks},
  author = {Zhenru Lin and Jiawen Tao and Yang Yuan and Andrew Chi-Chih Yao},
  year   = {2025},
  eprint = {2506.18781},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.18781v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.18781.svg)](https://codeofpaper.com/paper/2506.18781)

Discussion