Quantitative Clustering in Mean-Field Transformer Models
Machine Learning
2026-05-12 v3 Analysis of PDEs
Dynamical Systems
Machine Learning
Abstract
The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to the synchronization phenomenon in Kuramoto models. In this work, we investigate the long-time clustering of mean-field transformer models. More precisely, under suitable assumptions on the transformer model parameters, we establish that any suitably regular mean-field initialization synchronizes exponentially fast to a Dirac point mass, with explicit quantitative convergence rates.
Keywords
Cite
@article{arxiv.2504.14697,
title = {Quantitative Clustering in Mean-Field Transformer Models},
author = {Shi Chen and Zhengjiang Lin and Yury Polyanskiy and Philippe Rigollet},
journal= {arXiv preprint arXiv:2504.14697},
year = {2026}
}
Comments
50 pages, 4 figures; We have updated the introduction and added sketches of the proofs of the main theorems