Related papers: Homogenized Transformers
We introduce a family of stochastic models motivated by the study of nonequilibrium steady states of fluid equations. These models decompose the deterministic dynamics of interest into fundamental building blocks, i.e., minimal vector…
This paper studies homogenization of stochastic differential systems. The standard example of this phenomenon is the small mass limit of Hamiltonian systems. We consider this case first from the heuristic point of view, stressing the role…
Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the origin of these phenomena and their implications for…
We study the equilibrium glassy behavior of a multimode random laser model with nonlinear four-body quenched disordered interactions and a global smoothed-cubic constraint on mode intensities. This constraint, which provides a more…
We formulate a data-driven method for constructing finite volume discretizations of a dynamical system's underlying Continuity / Fokker-Planck equation. A method is employed that allows for flexibility in partitioning state space,…
In this paper, we develop a theoretical framework for nonlinear stochastic optimal control problems with optimal stopping by establishing a density-based deterministic representation of the underlying diffusion. For state-independent…
For a class of interacting particle systems in continuous space, we show that finite-volume approximations of the bulk diffusion matrix converge at an algebraic rate. The models we consider are reversible with respect to the Poisson…
In this paper, we study stochastic homogenization of a coupled diffusion-reaction system. The diffusion-reaction system is coupled to stochastic differential equations, which govern the changes in the media properties. Though homogenization…
We consider a suspension of active rigid particles (swimmers) in a steady Stokes flow, where particles are distributed according to a stationary ergodic random process, and we study its homogenization in the macroscopic limit. A key point…
Self-oscillation is a phenomenon studied across many scientific disciplines, including the engineering of efficient heat engines and electric generators. We investigate the single electron shuttle, a model nano-scale system that exhibits a…
The long time dynamics of large particles trapped in two inhomogeneous turbulent shear flows is studied experimentally. Both flows present a common feature, a shear region that separates two colliding circulations, but with different…
Transformers have emerged as a powerful neural network architecture capable of tackling a wide range of learning tasks. In this work, we provide a theoretical analysis of their ability to automatically extract structure from data in an…
Multiscale problems can usually be approximated through numerical homogenization by an equation with some effective parameters that can capture the macroscopic behavior of the original system on the coarse grid to speed up the simulation.…
We study the N-dependence of the thermodynamical variables and the dynamical behavior of the well-known Hamiltonian Mean Field model. Microcanonical analysis revealed a thermodynamic limit which defers from the a priory traditional…
A class of stochastic individual-based models, written in terms of coupled velocity jump processes, is presented and analysed. This modelling approach incorporates recent experimental findings on behaviour of locusts. It exhibits nontrivial…
Complex-valued signals encode both amplitude and phase, yet most deep models treat attention as real-valued correlation, overlooking interference effects. We introduce the Holographic Transformer, a physics-inspired architecture that…
Direct numerical simulations of mechanical metamaterials are prohibitively expensive due to the separation of scales between the lattice and the macrostructural size. Hence, multiscale continuum analysis plays a pivotal role in the…
We consider a nonlinear drift-diffusion system for multiple charged species in a porous medium in 2D and 3D with periodic microstructure. The system consists of a transport equation for the concentration of the species and Poisson's…
The training and generalization dynamics of the Transformer's core mechanism, namely the Attention mechanism, remain under-explored. Besides, existing analyses primarily focus on single-head attention. Inspired by the demonstrated benefits…
Multi-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer. Recent work has shown, however, that a large proportion of the heads in…