Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning
Abstract
Post-training is routinely evaluated through aggregate benchmark scores that treat multi-hop reasoning as a single capability -- as if a model that answers more questions correctly must be better at assembling facts. We show that this assumption can be misleading: recipes with statistically indistinguishable atomic knowledge produce composition behaviour separated by over 40 percentage points, a phenomenon we call composition collapse: the systematic failure to assemble stably-known facts into chains, invisible to aggregate metrics. We introduce a double-gate protocol that changes the estimand from an aggregate compositionality gap to residual composition failure conditioned on stable atomic access, decomposing post-training gains into three independent channels: atomic stability, residual composition, and critical depth. On a benchmark of temporal factual chains spanning depths 2--11 across four post-training recipes, this decomposition reveals that post-training objectives shift composition capability in directions that aggregate metrics mask, and suggests that claims about multi-hop reasoning improvement should be accompanied by atomic-gate-controlled composition metrics. Diagnostic probes further show that a substantial share of measured composition failure reflects generation-time computation constraints rather than permanent inability to compose.
Cite
@article{arxiv.2605.26789,
title = {Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning},
author = {Zhe Yu and Wenpeng Xing and Yunzhao Wei and Jie Chen and Hongzhi Wang and Xuyang Teng and Meng Han},
journal= {arXiv preprint arXiv:2605.26789},
year = {2026}
}