We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes -- including Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT -- revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying Peri-LN as a particularly effective choice.
@article{arxiv.2510.22026,
title = {Normalization in Attention Dynamics},
author = {Nikita Karagodin and Shu Ge and Yury Polyanskiy and Philippe Rigollet},
journal= {arXiv preprint arXiv:2510.22026},
year = {2025}
}
Comments
39th Conference on Neural Information Processing Systems (NeurIPS 2025), 23 pages