While humans naturally develop theory of mind (ToM), the capability to understand other people's mental states and beliefs, state-of-the-art large language models (LLMs) underperform on simple ToM benchmarks. We posit that we can extend our understanding of LLMs' ToM abilities by evaluating key human ToM precursors−perception inference and perception-to-belief inference−in LLMs. We introduce two datasets, Percept-ToMi and Percept-FANToM, to evaluate these precursory inferences for ToM in LLMs by annotating characters' perceptions on ToMi and FANToM, respectively. Our evaluation of eight state-of-the-art LLMs reveals that the models generally perform well in perception inference while exhibiting limited capability in perception-to-belief inference (e.g., lack of inhibitory control). Based on these results, we present PercepToM, a novel ToM method leveraging LLMs' strong perception inference capability while supplementing their limited perception-to-belief inference. Experimental results demonstrate that PercepToM significantly enhances LLM's performance, especially in false belief scenarios.
@article{arxiv.2407.06004,
title = {Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models},
author = {Chani Jung and Dongkwan Kim and Jiho Jin and Jiseon Kim and Yeon Seonwoo and Yejin Choi and Alice Oh and Hyunwoo Kim},
journal= {arXiv preprint arXiv:2407.06004},
year = {2024}
}