English

GPT-4 Can't Reason

Computation and Language 2023-08-11 v2

Abstract

GPT-4 was released in March 2023 to wide acclaim, marking a very substantial improvement across the board over GPT-3.5 (OpenAI's previously best model, which had powered the initial release of ChatGPT). However, despite the genuinely impressive improvement, there are good reasons to be highly skeptical of GPT-4's ability to reason. This position paper discusses the nature of reasoning; criticizes the current formulation of reasoning problems in the NLP community, as well as the way in which LLM reasoning performance is currently evaluated; introduces a small collection of 21 diverse reasoning problems; and performs a detailed qualitative evaluation of GPT-4's performance on those problems. Based on this analysis, the paper concludes that, despite its occasional flashes of analytical brilliance, GPT-4 at present is utterly incapable of reasoning.

Keywords

Cite

@article{arxiv.2308.03762,
  title  = {GPT-4 Can't Reason},
  author = {Konstantine Arkoudas},
  journal= {arXiv preprint arXiv:2308.03762},
  year   = {2023}
}
R2 v1 2026-06-28T11:50:09.033Z