Related papers: Spectral Selection in Symmetric Self-Attention Dyn…
Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process remains unclear. We present a mathematical framework to…
The aim of this paper is to provide a mathematical analysis of transformer architectures using a self-attention mechanism with layer normalization. In particular, observed patterns in such architectures resembling either clusters or uniform…
In machine learning, a self-attention dynamics is a continuous-time multiagent-like model of the attention mechanisms of transformers. In this paper we show that such dynamics is related to a multiagent version of the Oja flow, a dynamical…
Self-propelled particles can navigate complex environments, including viscous fluid interfaces with curved geometries. In this work, we study the emergent dynamics of a suspension of self-propelled particles confined to a stationary curved…
Transformer self-attention can be interpreted as a gradient flow on the unit sphere, in which tokens evolve under softmax interaction potentials and tend to form clusters. While prior work has established clustering behavior for single-head…
This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into sequence context.…
The self-attention mechanism prevails in modern machine learning. It has an interesting functionality of adaptively selecting tokens from an input sequence by modulating the degree of attention localization, which many researchers speculate…
We characterize the pre-softmax attention matrix $\mathbf{QK^\top}$ in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric…
The self-attention mechanism, now central to deep learning architectures such as Transformers, is a modern instance of a more general computational principle: learning and using pairwise affinity matrices to control how information flows…
We give a complete study of the asymptotic behavior of a simple model of alignment of unit vectors, both at the level of particles , which corresponds to a system of coupled differential equations, and at the continuum level, under the form…
We provide the detailed asymptotic behavior for first-order aggregation models of heterogeneous oscillators. Due to the dissimilarity of natural frequencies, one could expect that all relative distances converge to definite positive value…
The attention mechanism within the transformer architecture enables the model to weigh and combine tokens based on their relevance to the query. While self-attention has enjoyed major success, it notably treats all queries $q$ in the same…
Recently, a new line of works has emerged to understand and improve self-attention in Transformers by treating it as a kernel machine. However, existing works apply the methods for symmetric kernels to the asymmetric self-attention,…
We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phenomena in deep learning. We identify a consistent bias in…
A continuum model for self-organized dynamics is numerically investigated. The model describes systems of particles subject to alignment interaction and short-range repulsion. It consists of a non-conservative hyperbolic system for the…
The problem of finite-dimensional asymptotics of infinite-dimensional dynamic systems is studied. A non-linear kinetic system with conservation of supports for distributions has generically finite-dimensional asymptotics. Such systems are…
The control of wave scattering in complex non-Hermitian settings is an exciting subject -- often challenging the creativity of researchers and stimulating the imagination of the public. Successful outcomes include invisibility cloaks,…
In recent years, Transformer-based auto-attention mechanisms have been successfully applied to the analysis of a variety of context-reliant data types, from texts to images and beyond, including data from non-Euclidean geometries. In this…
In this paper we investigate spectral flow symmetry in asymptotically flat spacetimes both from a gravity as well as a putative dual quantum field theory perspective. On the gravity side we consider models in Einstein gravity and…
We study the evolution of radiating and viscous fluid spheres assuming an additional homothetic symmetry on the spherically simmetric space--time. We match a very simple solution to the symmetry equations with the exterior one (Vaidya). We…