English

11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis

Computation and Language 2025-08-28 v1 Computer Vision and Pattern Recognition Machine Learning

Abstract

For human cognitive process, spatial reasoning and perception are closely entangled, yet the nature of this interplay remains underexplored in the evaluation of multimodal large language models (MLLMs). While recent MLLM advancements show impressive performance on reasoning, their capacity for human-like spatial cognition remains an open question. In this work, we introduce a systematic evaluation framework to assess the spatial reasoning abilities of state-of-the-art MLLMs relative to human performance. Central to our work is 11Plus-Bench, a high-quality benchmark derived from realistic standardized spatial aptitude tests. 11Plus-Bench also features fine-grained expert annotations of both perceptual complexity and reasoning process, enabling detailed instance-level analysis of model behavior. Through extensive experiments across 14 MLLMs and human evaluation, we find that current MLLMs exhibit early signs of spatial cognition. Despite a large performance gap compared to humans, MLLMs' cognitive profiles resemble those of humans in that cognitive effort correlates strongly with reasoning-related complexity. However, instance-level performance in MLLMs remains largely random, whereas human correctness is highly predictable and shaped by abstract pattern complexity. These findings highlight both emerging capabilities and limitations in current MLLMs' spatial reasoning capabilities and provide actionable insights for advancing model design.

Keywords

Cite

@article{arxiv.2508.20068,
  title  = {11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis},
  author = {Chengzu Li and Wenshan Wu and Huanyu Zhang and Qingtao Li and Zeyu Gao and Yan Xia and José Hernández-Orallo and Ivan Vulić and Furu Wei},
  journal= {arXiv preprint arXiv:2508.20068},
  year   = {2025}
}

Comments

9 pages, 4 figures (22 pages, 7 figures, 7 tables including references and appendices)

R2 v1 2026-07-01T05:08:49.004Z