Related papers: Homogenized Transformers
We discuss the dynamics and thermodynamics of systems with long-range interactions. We contrast the microcanonical description of an isolated Hamiltonian system to the canonical description of a stochastically forced Brownian system. We…
In this paper, we study the evolution of tokens through the depth of encoder-only transformer models at inference time by modeling them as a system of particles interacting in a mean-field way and studying the corresponding dynamics. More…
Standard transformers entangle all computation in a single residual stream, obscuring which components perform which functions. We introduce the Dual-Stream Transformer, which decomposes the residual stream into two functionally distinct…
We present a unified framework, with quantitative estimates, for deterministic interacting particle systems whose pairwise interactions may depend on heterogeneous labels. Heterogeneity is kept at every level by adding a frozen label…
We consider an infinite-dimensional stochastic clustering model on $\mathbb{R}$. In discrete time, each point of a unit-intensity simple point process moves halfway toward either of its left or right neighbors, chosen uniformly at random.…
We consider a stochastic Hodgkin-Huxley model driven by a periodic signal as model for the membrane potential of a pyramidal neuron. The associated five dimensional diffusion process is a time inhomogeneous highly degenerate diffusion for…
The aim of our work is to provide a simple homogenization and discrete-to-continuum procedure for energy driven problems involving stochastic rapidly-oscillating coefficients. Our intention is to extend the periodic unfolding method to the…
We consider a system of differential equations in a fast long range dependent random environment and prove a homogenization theorem involving multiple scaling constants. The effective dynamics solves a rough differential equation, which is…
We study the dynamics of an inertial particle coupled to forcing, dissipation, and noise in the small mass limit. We derive an expression for the limiting (homogenized) joint distribution of the position and (scaled) velocity degrees of…
We consider the self-attention model - an interacting particle system on the unit sphere, which serves as a toy model for Transformers, the deep neural network architecture behind the recent successes of large language models. We prove the…
We study the relaxation dynamics of strongly interacting quantum systems that display a kind of many-body localization in spite of their translation-invariant Hamiltonian. We show that dynamics starting from a random initial configuration…
We consider the overdamped dynamics of a paradigmatic long-range system of particles residing on the sites of a one-dimensional lattice, in the presence of thermal noise. The internal degree of freedom of each particle is a periodic…
We consider a diffusion process with coefficients that are periodic outside of an "interface region" of finite thickness. The question investigated in this article is the limiting long time/large scale behavior of such a process under…
Mixing in open quantum systems is often summarized by a single worst-case time, even though that benchmark can be set by exponentially rare initial states. We show that for broad unstructured ensembles the nonlinear trace-distance…
It is well known that the classical energetically consistent micropolar model has limits in simulating the frequency band structure of packed granular materials (see Merkel et al., 2011). It is here shown that if a standard continualization…
We study the large-depth limit of transformers trained with AdamW, by modelling the hidden-state dynamics as an interacting particle system (IPS) coupled through the attention mechanism. Under appropriate scaling of the attention heads, we…
In {\em{Holm}, Proc. Roy. Soc. A 471 (2015)} stochastic fluid equations were derived by employing a variational principle with an assumed stochastic Lagrangian particle dynamics. Here we show that the same stochastic Lagrangian dynamics…
We perform the periodic homogenization (i.e. $\eps\to 0$) of the non-stationary Stokes-Nernst-Planck-Poisson system using two-scale convergence, where $\eps$ is a suitable scale parameter. The objective is to investigate the influence of…
We investigate the time-dependent, coherent, and dissipative dynamics of bound particles in single multilevel quantum dots in the presence of sequential tunnelling transport. We focus on the nonequilibrium regime where several channels are…
We examine the classic problem of homogeneous nucleation and growth by deriving and analyzing a fully discrete stochastic master equation. Upon comparison with results obtained from the corresponding mean-field Becker-D\"{o}ring equations…