Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worlds
Abstract
A fundamental challenge in reasoning is navigating hypothetical, counterfactual worlds where logic may conflict with ingrained knowledge. We investigate this frontier for Large Language Models (LLMs) by asking: Can LLMs reason logically when the context contradicts their parametric knowledge? To facilitate a systematic analysis, we first introduce CounterLogic, a benchmark specifically designed to disentangle logical validity from knowledge alignment. Evaluation of 11 LLMs across six diverse reasoning datasets reveals a consistent failure: model accuracy plummets by an average of 14% in counterfactual scenarios compared to knowledge-aligned ones. We hypothesize that this gap stems not from a flaw in logical processing, but from an inability to manage the cognitive conflict between context and knowledge. Inspired by human metacognition, we propose a simple yet powerful intervention: Flag & Reason (FaR), where models are first prompted to flag potential knowledge conflicts before they reason. This metacognitive step is highly effective, narrowing the performance gap to just 7% and increasing overall accuracy by 4%. Our findings diagnose and study a critical limitation in modern LLMs' reasoning and demonstrate how metacognitive awareness can make them more robust and reliable thinkers.
Cite
@article{arxiv.2505.22318,
title = {Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worlds},
author = {Anish R Joishy and Ishwar B Balappanawar and Vamshi Krishna Bonagiri and Manas Gaur and Krishnaprasad Thirunarayan and Ponnurangam Kumaraguru},
journal= {arXiv preprint arXiv:2505.22318},
year = {2026}
}