相关论文: Attention's forward pass and Frank-Wolfe

Large Language Models (LLMs) exhibit exceptional proficiency in handling extensive context windows in natural language. Nevertheless, the quadratic scaling of attention computation relative to sequence length creates substantial efficiency…

Transformers are state-of-the-art in a wide range of NLP tasks and have also been applied to many real-world products. Understanding the reliability and certainty of transformer model predictions is crucial for building trustable machine…

We develop a new numerical method for thin plates falling in inviscid fluid that allows for leading-edge vortex shedding. The inclusion of leading-edge shedding restores physical dynamics to vortex-sheet models of falling bodies, and for…

Transformer-based models have achieved remarkable success across a wide range of domains, yet our understanding of their training dynamics remains limited. In this work, we identify a recurrent focus-dilution cycle in attention learning and…

Tensor Attention extends traditional attention mechanisms by capturing high-order correlations across multiple modalities, addressing the limitations of classical matrix-based attention. Meanwhile, Rotary Position Embedding…

We develop an algorithmic framework for solving convex optimization problems using no-regret game dynamics. By converting the problem of minimizing a convex function into an auxiliary problem of solving a min-max game in a sequential…

We prove that the block-coordinate Frank-Wolfe (BCFW) algorithm converges with state-of-the-art rates in both convex and nonconvex settings under a very mild "block-iterative" assumption. This appears to be the first result on BCFW…

We analyse a coupled 3D-2D model with a free fluid governed by Stokes flow in the bulk and a poroelastic plate described by the Biot-Kirchhoff equations on the surface. Assuming the form of a double perturbed saddle-point problem, the…

We introduce Monte-Carlo Attention (MCA), a randomized approximation method for reducing the computational cost of self-attention mechanisms in Transformer architectures. MCA exploits the fact that the importance of each token in an input…

Maximal 't Hooft loops are studied in SO(3) lattice gauge theory at finite temperature T. Tunneling barriers among twist sectors causing loss of ergodicity for local update algorithms are overcome through parallel tempering, enabling us to…

We consider the dynamics of local entropy and nearest neighbor mutual information of a 1-D lattice of qubits via the repeated application of nearest neighbor CNOT quantum gates. This is a quantum version of a cellular automaton. We analyze…

Slow relaxation and glassiness have been the focus of extensive research attention, along with popular and technological interest, for many decades. While much understanding has been attained through mean-field and mode-coupling models,…

We investigate the thermodynamic and structural properties of divalent patchy hard rods confined to a one-dimensional channel by modeling the bonding sites as attractive square-well (SW) patches located at the rod tips. The zero-range…

Despite the popularity of the Transformer architecture, the standard algorithm for computing Attention suffers from quadratic time complexity in context length $n$. Alman and Song [NeurIPS 2023] showed that when the head dimension $d =…

In a Hilbert setting, we introduce a new dynamical system and associated algorithms for solving monotone inclusions by rapid methods. Given a maximal monotone operator $A$, the evolution is governed by the time dependent operator $I -(I +…

Linear attention has attracted interest as a computationally efficient approximation to softmax attention, especially for long sequences. Recent studies have explored distilling softmax attention in pre-trained Transformers into linear…

In a Hilbert space setting H, for convex optimization, we analyze the fast convergence properties as t tends to infinity of the trajectories generated by a third-order in time evolution system. The function f to minimize is supposed to be…

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight…

We analyse a $2+1$ dimensional model with charged, relativistic fermions interacting through a four-Fermi term. Taking advantage of its large-$N$ renormalizability, the various phases of this model are studied at finite temperature and…

Length-dependent logit rescaling is widely used to stabilize long-context self-attention, but existing analyses and methods suggest conflicting inverse-temperature laws for the context length $n$, ranging from $(\log n)^{1/2}$ to $\log n$…