Existing LLMs Are Not Self-Consistent For Simple Tasks
Abstract
Large Language Models (LLMs) have grown increasingly powerful, yet ensuring their decisions remain transparent and trustworthy requires self-consistency -- no contradictions in their internal reasoning. Our study reveals that even on simple tasks, such as comparing points on a line or a plane, or reasoning in a family tree, all smaller models are highly inconsistent, and even state-of-the-art models like DeepSeek-R1 and GPT-o4-mini are not fully self-consistent. To quantify and mitigate these inconsistencies, we introduce inconsistency metrics and propose two automated methods -- a graph-based and an energy-based approach. While these fixes provide partial improvements, they also highlight the complexity and importance of self-consistency in building more reliable and interpretable AI. The code and data are available at https://github.com/scorpio-nova/llm-self-consistency.
Cite
@article{arxiv.2506.18781,
title = {Existing LLMs Are Not Self-Consistent For Simple Tasks},
author = {Zhenru Lin and Jiawen Tao and Yang Yuan and Andrew Chi-Chih Yao},
journal= {arXiv preprint arXiv:2506.18781},
year = {2025}
}
Comments
10 pages, 6 figures