相关论文: Dynamic metastability in the self-attention model
Metastability, i.e., partial relaxation to long-lived, quasi-stationary states before true asymptotic equilibrium sets in, emerges ubiquitously in classical and quantum dynamical systems as a result of timescales separation. In open quantum…
We study a catching-up algorithm for a class of differential inclusions driven by maximal monotone operators with continuous perturbations. Using a decomposition of the monotone operator into the closed convex hull of its single-valued part…
Self-organization is ubiquitous in nature and mind. However, machine learning and theories of cognition still barely touch the subject. The hurdle is that general patterns are difficult to define in terms of dynamical equations and…
Actively propelled particles undergoing dissipative collisions are known to develop a state of spatially distributed coherently moving clusters. For densities larger than a characteristic value clusters grow in time and form a stationary…
We study self-attention dynamics on the unit sphere as an interacting particle system arising from an idealized Transformer-type update. Under a symmetry assumption on weight matrices given by $Q^\top K=V=V^\top$, the flow admits a…
We develop a finite-dimensional sensitivity framework for studying stability in learning systems whose states include representations, parameters, and update variables. The central object is the \emph{Learning Stability Profile}, a…
The transformer architecture and variants presented remarkable success across many machine learning tasks in recent years. This success is intrinsically related to the capability of handling long sequences and the presence of…
In this note we discuss metastability in a long-but-finite range disordered model for the glass transition. We show that relaxation is dominated by configuration belonging to metastable states and associate an in principle computable…
We study the energy flow of dissipative dynamics on infinite lattices, allowing the total energy to be infinite and considering formally gradient dynamics. We show that in spatial dimensions 1,2, the flow is for almost all times arbitrarily…
We show how the tracer motion of tagged, distinguishable particles can effectively describe transport in various homogeneous quantum many-body systems with constraints. We consider systems of spinful particles on a one-dimensional lattice…
Using agent-based simulations of self-propelled particles subject to short-range repulsion and nematic alignment we explore the dynamical phases of a dense active material confined to the surface of a sphere. We map the dynamical phase…
Self-attention is an attention mechanism that learns a representation by relating different positions in the sequence. The transformer, which is a sequence model solely based on self-attention, and its variants achieved state-of-the-art…
We present a derivation of a recently proposed theory for the time dependence of density fluctuations in stationary states of strongly interacting, athermal, self-propelled particles. The derivation consists of two steps. First, we start…
In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks. However, in gated linear attention models, this memory has a fixed…
A study of the gas dynamics of a dilute collection of the inelastically colliding hard spheres is presented. When diffusive processes are neglected the gas density blows up in a finite time. The blowup is the mathematical expression for one…
Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to improve performance, a gated attention model that leverages a…
A mode-coupling theory for the slow single-particle dynamics in fluids adsorbed in disordered porous media is derived, which complements previous work on the collective dynamics [V. Krakoviack, Phys. Rev. E 75, 031503 (2007)]. Its…
Rich out of equilibrium collective dynamics of strongly interacting large assemblies emerge in many areas of science. Some intriguing and not fully understood examples are the glassy arrest in atomic, molecular or colloidal systems,…
My goal is to study the dynamics of the Universe from a relational perspective based on the happening of events in temporal relation to each other and their respective points of reference. Accordingly, the flow of time was modeled as the…
The Transformer, a breakthrough architecture in artificial intelligence, owes its success to the attention mechanism, which utilizes long-range interactions in sequential data, enabling the emergent coherence between large language models…