Attention Sinks and Outliers in Attention Residuals
Abstract
We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel, they improve inter-layer routing flexibility but also exacerbate attention sinks, activation outliers, and the resulting degradation in inference stability and quantization robustness. OASIS addresses this issue by introducing a Softmax1-based null space and coupling token-level null evidence to depth routing through an inter-layer null signal, thereby reducing sink-dominated routing and improving structural robustness. Theoretically, we show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness. Experimentally, we compare OASIS against five baselines on three real-world datasets and observe consistent improvements in both attention sink and post-quantization performance. Notably, OASIS achieves an average reduction of 9.26% in maximum infinity norm and 2.60% in average kurtosis across the evaluated settings, while lowering perplexity by 75.85% under W8A8 and improving GSM8K Pass@1 by 12.42% under W4A4.
Cite
@article{arxiv.2605.17887,
title = {Attention Sinks and Outliers in Attention Residuals},
author = {Haozheng Luo and Haoran Dai and Shaoyang Zhang and Xi Chen and Eric Hanchen Jiang and Yijiang Li and Jingyuan Huang and Chenghao Qiu and Chenwei Xu and Zhenyu Pan and Haotian Zhang and Binghui Wang and Yan Chen},
journal= {arXiv preprint arXiv:2605.17887},
year = {2026}
}