Related papers: Quantitative Clustering in Mean-Field Transformer …
We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein…
In this paper, we study the evolution of tokens through the depth of encoder-only transformer models at inference time by modeling them as a system of particles interacting in a mean-field way and studying the corresponding dynamics. More…
We model the evolution of tokens within a deep stack of Transformer layers as a continuous-time flow on the unit sphere, governed by a mean-field interacting particle system, building on the framework introduced in (Geshkovski et al.,…
Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive…
Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at…
We study a network of finitely many interacting clusters where each cluster is a collection of globally coupled circle maps in the thermodynamic (or mean field) limit. The state of each cluster is described by a probability measure, and its…
We consider mean-field models for data--clustering problems starting from a generalization of the bounded confidence model for opinion dynamics. The microscopic model includes information on the position as well as on additional features of…
Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as…
The emergence of explosive synchronization has been reported as an abrupt transition in complex networks of first-order Kuramoto oscillators. In this Letter, we demonstrate that the nodes in a second-order Kuramoto model, perform a cascade…
This paper studies a mean field game formulation of the classical Kuramoto model for synchronization. Our model captures the diversity within the population by considering random intrinsic frequencies, which allows us to study the impact of…
This paper investigates the use of transformers to approximate the mean-field dynamics of interacting particle systems exhibiting collective behavior. Such systems are fundamental in modeling phenomena across physics, biology, and…
A striking clustering phenomenon in the antiferromagnetic Hamiltonian Mean-Field model has been previously reported. The numerically observed bicluster formation and stabilization is here fully explained by a non linear analysis of the…
The cluster synchronization is a very important characteristic for the higher harmonic coupling Kuramoto system. A novel transformation is provided, and it gives cluster synchronization by the periodic properties of the density function.…
This paper considers a mean-field model of $n$ interacting particles whose state space is the unit circle, a generalization of the classical Kuramoto model. Global synchronization is said to occur if after starting from almost any initial…
We develop, clarify and test various aspects of cluster methods dynamical mean field methods using a soluble toy model as a benchmark. We find that the Cellular Dynamical Mean Field Theory (C-DMFT) converges very rapidly and compare its…
The Kuramoto model, despite its popularity as a mean-field theory for many synchronization phenomenon of oscillatory systems, is limited to a first-order harmonic coupling of phases. For higher-order coupling, there only exists a…
The extended dynamical mean field theory has played an important role in the study of quantum phase transitions in heavy fermion systems. In order to incorporate the physics of unconventional superconductivity, we develop a cluster version…
Quantum cluster approaches offer new perspectives to study the complexities of macroscopic correlated fermion systems. These approaches can be understood as generalized mean-field theories. Quantum cluster approaches are non-perturbative…
A general expansion scheme based on the concept of linked cluster expansion from the theory of classical spin systems is constructed for models of interacting electrons. It is shown that with a suitable variational formulation of mean-field…
Simulations of extended quantum systems are typically performed by extrapolating results of a sequence of finite-system-size simulations to the thermodynamic limit. In the quantum Monte Carlo community, twist-averaging was pioneered as an…