相关论文: A multiscale analysis of mean-field transformers i…

Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at…

The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to the synchronization phenomenon in Kuramoto models. In this…

We model the evolution of tokens within a deep stack of Transformer layers as a continuous-time flow on the unit sphere, governed by a mean-field interacting particle system, building on the framework introduced in (Geshkovski et al.,…

Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive…

We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein…

Understanding how information propagates through Transformer models is a key challenge for interpretability. In this work, we study the effects of minimal token perturbations on the embedding space. In our experiments, we analyze the…

Ergodic quantum many-body systems undergoing unitary dynamics evolve towards increasingly entangled states characterized by an extensive scaling of entanglement entropy with system volume. At the other extreme, quantum systems repeatedly…

The competition between unitary quantum dynamics and dissipative stochastic effects, as emerging from continuous-monitoring processes, can culminate in measurement-induced phase transitions. Here, a many-body system abruptly passes, when…

We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer model with MultiLayer Perceptron (MLP) blocks to a continuous-time stochastic interacting particle system. We also identify the…

We use a mean-field-based transformer model to theoretically investigate how auxiliary variables, such as positional encoding, prevent mode collapse of self-attention mechanisms. The use of mean-field transformers to analyze the properties…

This paper investigates the use of transformers to approximate the mean-field dynamics of interacting particle systems exhibiting collective behavior. Such systems are fundamental in modeling phenomena across physics, biology, and…

The role of thermodynamics in the evolution of systems evolving under purely gravitational forces is not completely established. Both the infinite range and singularity in the Newtonian force law preclude the use of standard techniques.…

We investigate the relationship between the geometry of token embeddings and their role in the next token prediction within transformer models. An important aspect of this connection uses the notion of empirical measure, which encodes the…

This paper investigates the dynamical properties of tokens in pre-trained Transformer models and explores their application to improving Transformers. To this end, we analyze the dynamical system governing the continuous-time limit of the…

Repeated measurements can induce entanglement phase transitions in the dynamics of quantum systems. Interacting models, both chaotic and integrable, generically show a stable volume-law entangled phase at low measurement rates which…

We consider a large number $N$ of quantum particles coupled via a mean field interaction to another quantum system (reservoir). Our main result is an expansion for the averages of observables, both of the particles and of the reservoir, in…

We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads, as at initialization of training. Viewing depth as a time variable, the residual stream defines a…

This paper studies large deviations of a ``fully coupled" finite state mean-field interacting particle system in a fast varying environment. The empirical measure of the particles evolves in the slow time scale and the random environment…

The in-plane magnetoconductance of the strongly interacting two-dimensional electron system in a silicon MOSFET (metal-oxide-semiconductor-field-effect transistor) exhibits an unmistakeable kink at a well-defined electron density, $n_k$.…

The evolution of a quantum system subject to measurements can be described by stochastic quantum trajectories of pure states. Instead, the ensemble average over trajectories is a mixed state evolving via a master equation. Both descriptions…