Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes are available at https://github.com/NEUIR/Uncode.
@article{arxiv.2508.13021,
title = {Empirical Analysis of Decoding Biases in Masked Diffusion Models},
author = {Pengcheng Huang and Tianming Liu and Zhenghao Liu and Yukun Yan and Shuo Wang and Tong Xiao and Zulong Chen and Maosong Sun},
journal= {arXiv preprint arXiv:2508.13021},
year = {2026}
}