Related papers: Emergence of meta-stable clustering in mean-field …
We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein…
Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at…
In this paper, we study the evolution of tokens through the depth of encoder-only transformer models at inference time by modeling them as a system of particles interacting in a mean-field way and studying the corresponding dynamics. More…
Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the attention function, which learns dependencies between tokens and…
The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to the synchronization phenomenon in Kuramoto models. In this…
The aim of this paper is to provide a mathematical analysis of transformer architectures using a self-attention mechanism with layer normalization. In particular, observed patterns in such architectures resembling either clusters or uniform…
Transformers have become the dominant architecture in modern machine learning, yet the theoretical understanding of their training dynamics remains limited. This paper develops a rigorous mathematical framework for analyzing gradient-based…
We show that the standard discrete update rule of transformer layers can be naturally interpreted as a forward Euler discretization of a continuous dynamical system. Our Transformer Flow Approximation Theorem demonstrates that, under…
Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive…
Transformer self-attention can be interpreted as a gradient flow on the unit sphere, in which tokens evolve under softmax interaction potentials and tend to form clusters. While prior work has established clustering behavior for single-head…
Deep learning paradigms, such as PINNs and neural operators, have significantly advanced the solving of PDEs. However, they often struggle to capture the continuous integral nature of physical systems, relying either on pointwise residuals…
In conflict with standard notions of thermodynamics, mesoscopically-sized inclusions (``clusters'') of a solute-rich liquid have been observed in equilibrated solutions of proteins and other molecules. According to a complexation scenario…
The formation of patterns in driven systems has been studied extensively, and their emergence can be connected to a fine balance of instabilities and stabilization mechanisms. While the early phase of pattern formation can be understood on…
A pathwise large deviation principle in the Wasserstein topology and a pathwise central limit theorem are proved for the empirical measure of a mean-field system of interacting diffusions. The coefficients are path-dependent. The framework…
We consider the self-attention model - an interacting particle system on the unit sphere, which serves as a toy model for Transformers, the deep neural network architecture behind the recent successes of large language models. We prove the…
We study interacting particle systems of Kuramoto-type. Our focus is on the dynamical relation between the partial differential equation (PDE) arising in the continuum limit (CL) and the one obtained in the mean-field limit (MFL). Both…
We study the Wasserstein gradient flow of semi-discrete energies in the space of probability measures, that is functionals depending on two measures-one being an absolutely continuous density and the other an atomic measure. These energies…
We discuss metastable states in the mean-field version of the strong coupling BCS-model and study the evolution of a superconducting equilibrium state subjected to a dynamical semi-group with Lindblad generator in detailed balance w.r.t.…
This Article presents a nonequilibrium thermodynamic theory for the mean-field precipitation, aggregation and pattern formation of colloidal clusters. A variable gradient energy coefficient and the arrest of particle diffusion upon…
Many-body long-range interacting systems can remain approximately in a quasi-stationary state far-from-thermodynamic equilibrium. These states are typically characterized by a pair of counter-propagating density clusters, or by a single…