Indirect Attention: Turning Context Misalignment into a Feature
Abstract
The attention mechanism has become a cornerstone of modern deep learning architectures, where keys and values are typically derived from the same underlying sequence or representation. This work explores a less conventional scenario, when keys and values originate from different sequences or modalities. Specifically, we first analyze the attention mechanism's behavior under noisy value features, establishing a critical noise threshold beyond which signal degradation becomes significant. Furthermore, we model context (key, value) misalignment as an effective form of structured noise within the value features, demonstrating that the noise induced by such misalignment can substantially exceed this critical threshold, thereby compromising standard attention's efficacy. Motivated by this, we introduce Indirect Attention, a modified attention mechanism that infers relevance indirectly in scenarios with misaligned context. We evaluate the performance of Indirect Attention across a range of synthetic tasks and real world applications, showcasing its superior ability to handle misalignment.
Cite
@article{arxiv.2509.26015,
title = {Indirect Attention: Turning Context Misalignment into a Feature},
author = {Bissmella Bahaduri and Hicham Talaoubrid and Fangchen Feng and Zuheng Ming and Anissa Mokraoui},
journal= {arXiv preprint arXiv:2509.26015},
year = {2025}
}