English

Large Vision-Language Models Get Lost in Attention

Artificial Intelligence 2026-05-08 v1 Computer Vision and Pattern Recognition

Abstract

Despite the rapid evolution of training paradigms, the decoder backbone of large vision--language models (LVLMs) remains fundamentally rooted in the residual-connection Transformer architecture. Therefore, deciphering the distinct roles of internal modules is critical for understanding model mechanics and guiding architectural optimization. While prior statistical approaches have provided valuable attribution-based insights, they often lack a unified theoretical basis. To bridge this gap, we propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates. Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-preserving operator focused on reconfiguration, whereas FFNs serve as subspace-expanding operators driving semantic innovation. Strikingly, further experiments demonstrate that replacing learned attention weights with predefined values (e.g., Gaussian noise) yields comparable or even superior performance across a majority of datasets relative to vanilla models. These results expose severe misallocation and redundancy in current mechanisms, suggesting that state-of-the-art LVLMs effectively ``get lost in attention'' rather than efficiently leveraging visual context.

Keywords

Cite

@article{arxiv.2605.05668,
  title  = {Large Vision-Language Models Get Lost in Attention},
  author = {Gongli Xi and Ye Tian and Mengyu Yang and Huahui Yi and Liang Lin and Xiaoshuai Hao and Kun Wang and Wendong Wang},
  journal= {arXiv preprint arXiv:2605.05668},
  year   = {2026}
}

Comments

25 pages, 10 figures. Accepted by ICML 2026

R2 v1 2026-07-01T12:54:05.939Z