Related papers: Homogenized Transformers
Systems with conserved currents driven by reservoirs at the boundaries offer an opportunity for a general analytic study that is unparalleled in more general out of equilibrium systems. The evolution of coarse-grained variables is governed…
While modern representation learning relies heavily on global error signals, decentralized algorithms driven by local interactions offer a fundamental distributed alternative. However, the macroscopic convergence properties of these…
Dynamic heterogeneity has often been modeled by assuming that a single-particle observable, fluctuating at a molecular scale, is influenced by its coupling to environmental variables fluctuating on a second, perhaps slower, time scale.…
We show that a constant number of self-attention layers can efficiently simulate, and be simulated by, a constant number of communication rounds of Massively Parallel Computation. As a consequence, we show that logarithmic depth is…
We present a first-principles formalism for studying dynamical heterogeneities in glass forming liquids. Based on the Non-Equilibrium Self-Consistent Generalized Langevin Equation theory, we were able to describe the time-dependent local…
We study homogenization of a locally periodic two-scale dual-continuum system where each continuum interacts with the other. Equations for each continuum are written separately with interaction terms (exchange terms) added. The…
We consider a one-dimensional diffusion process with coefficients that are periodic outside of a finite 'interface region'. The question investigated in this article is the limiting long time / large scale behaviour of such a process under…
In this work, we analyze various scaling limits of the training dynamics of transformer models in the feature learning regime. We identify the set of parameterizations that admit well-defined infinite width and depth limits, allowing the…
We perform the homogenization process avoiding the necessity of testing the weak formulation of the initial and homogenized systems by corresponding weak solutions. We show that the stress tensor for homogenized problem depends on the…
We study a model for a dilute suspension of rod-like particles swimming at constant velocity in a Stokes flow. As the translational diffusivity of the particles decreases, a two-dimensional uniform concentration of randomly aligned…
In this paper, we study the homogenization of the distribution-dependent stochastic abstract fluid models by combining the $two\!-\!scale$ convergence and martingale representative approach. A general framework of the homogenization…
Inspired by topological data analysis techniques, we introduce persistent homology observables and apply them in a geometric analysis of the dynamics of quantum field theories. As a prototype application, we consider data from a…
Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive…
We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective…
Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at…
In this paper, we derive new continuous time limits of the Utterance Selection Model (USM) for language change (Baxter et al., Phys. Rev. E {\bf 73}, 046118, 2006). This is motivated by the fact that the Fokker-Planck continuous time limit…
In this paper, we study the homogenization of a diffusion process with jumps, that is, Feller process generated by an integro-differential operator. This problem is closely related to the problem of homogenization of boundary value problems…
We develop a classical well-posedness and regularity theory on a finite connected weighted graph for an extended mean field game system, its associated master equation, and a Hamilton-Jacobi- Bellman equation on the probability simplex, all…
Interacting particle systems provide a fundamental framework for modeling collective behavior in biological, social, and physical systems. In many applications, stochastic perturbations are essential for capturing environmental variability…
We study the problem of homogenization for inertial particles moving in a time dependent random velocity field and subject to molecular diffusion. We show that, under appropriate assumptions on the velocity field, the large--scale,…