相关论文: Dynamic metastability in the self-attention model

The problem of metastability for a stochastic dynamics with a parallel updating rule is addressed in the Freidlin--Wentzel regime, namely, finite volume, small magnetic field, and small temperature. The model is characterized by the…

This model describes cluster aggregation in a stirred colloidal solution Interacting clusters compete for growth in this 'winner-takes-all' model; for finite assemblies, the largest cluster always wins, i.e. there is a uniform sediment. In…

The emergence of clustering and coarsening in crowded ensembles of self-propelled agents is studied using a lattice model in one-dimension. The persistent exclusion process, where particles move at directions that change randomly at a low…

Transport at microscopic length scales is essential in biological systems and various technologies, including microfluidics. Recent experiments achieved self-organized transport phenomena in microtubule active matter using light to modulate…

Despite the diversity of materials designated as active matter, virtually all active systems undergo a form of dynamic arrest when crowding and activity compete, reminiscent of the dynamic arrest observed in colloidal and molecular fluids…

The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by…

This paper continues the study of metastable behaviour in disordered mean field models initiated in [2], [3]. We consider the generalized Hopfield model with finitely many independent patterns $\xi_1,...,\xi_p$ where the patterns have…

We quantitatively characterize the metastability in a multi-phase lattice Boltzmann model. The structure factor of density fluctuations is theoretically obtained and numerically verified to a high precision, for all simulated wave-vectors…

We study an interacting high-density one-dimensional system of self-propelled particles described by the Active Ornstein-Uhlenbeck particle (AOUP) model where, even in the absence of alignment interactions, velocity and energy domains…

In this paper, we detail the relationship between convolutions and self-attention in natural language tasks. We show that relative position embeddings in self-attention layers are equivalent to recently-proposed dynamic lightweight…

Multi-Agent Reinforcement Learning involves agents that learn together in a shared environment, leading to emergent dynamics sensitive to initial conditions and parameter variations. A Dynamical Systems approach, which studies the evolution…

We study the dynamics of gradient flow for training a multi-head softmax attention model for in-context learning of multi-task linear regression. We establish the global convergence of gradient flow under suitable choices of initialization.…

We study dynamics of clustering in systems containing active particles that are immersed in an explicit solvent. For this purpose we have adopted a hybrid simulation method, consisting of molecular dynamics and multi-particle collision…

The phase behavior of stabilized dispersions of macromolecules is most easily described in terms of the effective interaction between the centers of mass of solute particles. For molecules like polymer chains, dendrimers, etc., the…

Transformers have demonstrated remarkable performance across diverse domains. The key component of Transformers is self-attention, which learns the relationship between any two tokens in the input sequence. Recent studies have revealed that…

The large-time behavior of solutions to Burgers equation with small viscosity is described using invariant manifolds. In particular, a geometric explanation is provided for a phenomenon known as metastability, which in the present context…

Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly for long sequences,…

This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into sequence context.…

Collective behavior of self-propelled particles is observed on a microscale for swimmers such as sperm and bacteria as well as for protein filaments in motility assays. The properties of such systems depend both on their dimensionality and…

Transformers have emerged as a powerful neural network architecture capable of tackling a wide range of learning tasks. In this work, we provide a theoretical analysis of their ability to automatically extract structure from data in an…