Related papers: Homogenized Transformers
Transformer self-attention can be interpreted as a gradient flow on the unit sphere, in which tokens evolve under softmax interaction potentials and tend to form clusters. While prior work has established clustering behavior for single-head…
We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer model with MultiLayer Perceptron (MLP) blocks to a continuous-time stochastic interacting particle system. We also identify the…
A microscopic heterogeneous system under random influence is considered. The randomness enters the system at physical boundary of small scale obstacles as well as at the interior of the physical medium. This system is modeled by a…
Learning reduced descriptions of chaotic many-body dynamics is fundamentally challenging: although microscopic equations are Markovian, collective observables exhibit strong memory and exponential sensitivity to initial conditions and…
We study the dynamics of a class of Hamiltonian systems with dissipation, coupled to noise, in a singular (small mass) limit. We derive the homogenized equation for the position degrees of freedom in the limit, including the presence of a…
Transformer-based models have demonstrated exceptional performance across diverse domains, becoming the state-of-the-art solution for addressing sequential machine learning problems. Even though we have a general understanding of the…
We study the long-time dynamics of two-dimensional linear Fokker-Planck equations driven by a drift that can be decomposed in the sum of a large shear component and the gradient of a regular potential depending on one spatial variable. The…
High-harmonic generation (HHG) in solids provides a powerful platform to probe ultrafast electron dynamics and interband--intraband coupling. However, disentangling the complex many-body contributions in the HHG spectrum remains…
We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein…
Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the attention function, which learns dependencies between tokens and…
We give an explicit stochastic Hamiltonian model of discontinuous unitary evolution for quantum spontaneous jumps like in a system of atoms in quantum optics, or in a system of quantum particles that interacts singularly with "bubbles"…
The diffusion of molecules in complex intracellular environments can be strongly influenced by spatial heterogeneity and stochasticity. A key challenge when modelling such processes using stochastic random walk frameworks is that negative…
This study deals with continuous limits of interacting one-dimensional diffusive systems, arising from stochastic distortions of discrete curves with various kinds of coding representations. These systems are essentially of a…
Transformers have revolutionized deep learning across various domains but understanding the precise token dynamics remains a theoretical challenge. Existing theories of deep Transformers with layer normalization typically predict that…
A variational coarse-graining framework for heterogeneous media is developed that allows for a seamless transition from the traditional static scenario to a arbitrary loading conditions, including inertia effects and body forces. The…
Fully resolving dynamics of materials with rapidly-varying features involves expensive fine-scale computations which need to be conducted on macroscopic scales. The theory of homogenization provides an approach to derive effective…
In recent years, transformer architectures have revolutionized the field of language processing, opening the door to previously unforeseen possibilities. However, from a theoretical point of view, the mathematical models proposed in the…
We study the homogenization problem for a system of stochastic differential equation with local time terms that models a multivariate diffusion in presence of semipermeable hyperplane interfaces with oblique penetration. We show that this…
A formalism for quantum many-body systems is proposed through a semiclassical treatment in phase space, allowing us to establish a stochastic thermodynamics incorporating quantum statistics. Specifically, we utilize a stochastic…
Viewing Transformers as interacting particle systems, we describe the geometry of learned representations when the weights are not time dependent. We show that particles, representing tokens, tend to cluster toward particular limiting…