相关论文: Dynamic metastability in the self-attention model

In this work, we introduce an information-theoretic approach for considering changes in dynamics of finitely dimensional open quantum systems governed by master equations. This experimentally motivated approach arises from considering how…

We model the evolution of tokens within a deep stack of Transformer layers as a continuous-time flow on the unit sphere, governed by a mean-field interacting particle system, building on the framework introduced in (Geshkovski et al.,…

The self-attention mechanism, a cornerstone of Transformer-based state-of-the-art deep learning architectures, is largely heuristic-driven and fundamentally challenging to interpret. Establishing a robust theoretical foundation to explain…

Metastability in open system dynamics describes the phenomena of initial relaxation to longlived metastable states before decaying to the asymptotic stable states. It has been predicted in continuous-time stochastic dynamics of both…

The self-attention mechanism (SAM) is widely used in various fields of artificial intelligence and has successfully boosted the performance of different models. However, current explanations of this mechanism are mainly based on intuitions…

Deep learning employs multi-layer neural networks trained via the backpropagation algorithm. This approach has achieved success across many domains and relies on adaptive gradient methods such as the Adam optimizer. Sequence modeling…

Phase transitions mark qualitative reorganizations of collective behavior, yet identifying their boundaries remains challenging whenever analytic solutions are absent and conventional simulations fail. Here we introduce learnability as a…

The element of metastability is incorporated in the fluctuating nonlinear hydrodynamic description of the mode coupling theory (MCT) of the liquid-glass transition. This is achieved through the introduction of the defect density variable…

Transformers are increasingly adopted for modeling and forecasting time-series, yet their internal mechanisms remain poorly understood from a dynamical systems perspective. In contrast to classical autoregressive and state-space models,…

Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation. In this work, we study learning a 1-layer self-attention model from a set of prompts and associated…

We consider the problem of learning the dynamics in the topology of time-evolving point clouds, the prevalent spatiotemporal model for systems exhibiting collective behavior, such as swarms of insects and birds or particles in physics. In…

In this paper we develop a metastability theory for a class of stochastic reaction-diffusion equations exposed to small multiplicative noise. We consider the case where the unperturbed reaction-diffusion equation features multiple…

We study the Glauber dynamics for heavy-tailed spin glasses, in which the couplings are in the domain of attraction of an $\alpha$-stable law for $\alpha\in (0,1)$. We show a sharp description of metastability on exponential timescales, in…

By means of the isoconfigurational method we calculate the change in the propensity for motion that the structure of a glass-forming system experiences during its relaxation dynamics. The relaxation of such a system has been demonstrated to…

We study the fundamental optimization principles of self-attention, the defining mechanism of transformers, by analyzing the implicit bias of gradient-based optimizers in training a self-attention layer with a linear decoder in binary…

The behavior of a driven granular gas in a container consisting of $M$ connected compartments is studied employing a microscopic kinetic model. After obtaining the governing equations for the occupation numbers and the granular temperatures…

The self-attention mechanism has significantly advanced the field of natural language processing, facilitating the development of advanced language-learning machines. Although its utility is widely acknowledged, the precise mechanisms of…

Attention mechanism has been extensively integrated within mainstream neural network architectures, such as Transformers and graph attention networks. Yet, its underlying working principles remain somewhat elusive. What is its essence? Are…

We establish a link between metastability and a discrete time-crystalline phase in a periodically driven open quantum system. The mechanism we highlight requires neither the system to display any microscopic symmetry nor the presence of…

We investigate the collective dynamics of self-propelled droplets, confined in a one dimensional micro-fluidic channel. On one hand, neighboring droplets align and form large trains of droplets moving in the same direction. On the other…