English

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

Software Engineering 2026-05-28 v2 Artificial Intelligence

Abstract

Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).

Cite

@article{arxiv.2603.24631,
  title  = {Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code},
  author = {Myeongsoo Kim and Dingmin Wang and Siwei Cui and Farima Farmahinifarahani and Terry Yue Zhuo and Shweta Garg and Baishakhi Ray and Rajdeep Mukherjee and Varun Kumar},
  journal= {arXiv preprint arXiv:2603.24631},
  year   = {2026}
}
R2 v1 2026-07-01T11:37:50.115Z