相关论文: Scaling Limits of Long-Context Transformers

Two dimensional condensed matter is realised in increasingly diverse forms that are accessible to experiment and of potential technological value. The properties of these systems are influenced by many length scales and reflect both generic…

We consider long-range Bernoulli bond percolation on the $d$-dimensional hierarchical lattice in which each pair of points $x$ and $y$ are connected by an edge with probability $1-\exp(-\beta\|x-y\|^{-d-\alpha})$, where $0<\alpha<d$ is…

The attention mechanism has become a cornerstone of modern deep learning architectures, where keys and values are typically derived from the same underlying sequence or representation. This work explores a less conventional scenario, when…

While linear attention reduces the quadratic complexity of standard Transformers to linear time, it often lags behind in expressivity due to the removal of softmax normalization. This omission eliminates \emph{global competition}, a…

Self-attention and masked self-attention are at the heart of Transformers' outstanding success. Still, our mathematical understanding of attention, in particular of its Lipschitz properties - which are key when it comes to analyzing…

High temperature expansions for the free energy, the susceptibility and the second correlation moment of the classical N-vector model [also known as the O(N) symmetric classical spin Heisenberg model or as the lattice O(N) nonlinear sigma…

We study the behavior of systems in which the interaction contains a long-range component that does not dominate the critical behavior. Such a component is exemplified by the van der Waals force between molecules in a simple liquid-vapor…

One persistent challenge in LLM research is the development of attention mechanisms that are able to generalise from training on shorter contexts to inference on longer contexts. We propose two conditions that we expect all effective long…

The finite size behavior of the susceptibility, Binder cumulant and some even moments of the magnetization of a fully finite O(n) cubic system of size L are analyzed and the corresponding scaling functions are derived within a…

Sparse Attention is a technique that approximates standard attention computation with sub-quadratic complexity. This is achieved by selectively ignoring smaller entries in the attention matrix during the softmax function computation.…

Transformer-based large language models (LLMs) excel in natural language processing tasks by capturing long-range dependencies through self-attention mechanisms. However, long-context modeling faces significant computational inefficiencies…

We uncover a universal scaling law governing the dispersion of collective attention and identify its underlying stochastic criticality. By analysing large-scale ensembles of Wikipedia page views, we find that the variance of logarithmic…

The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention…

Transformer language models have driven significant progress across various fields, including natural language processing and computer vision. A central component of these models is the self-attention (SA) mechanism, which learns rich…

We perform a comprehensive analysis of a collective decision-making model inspired by honeybee behavior. This model integrates individual exploration for option discovery and social interactions for information sharing, while also…

We propose a simple scaling theory describing critical effects at rounded meniscus osculation transitions which occur when the Laplace radius of a condensed macroscopic drop of liquid coincides with the local radius of curvature $R_w$ in a…

Prior work on input-token importance in auto-regressive transformers has relied on Softmax-normalized attention weights, which obscure the richer structure of pre-Softmax query-key logits. We introduce RCStat, a statistical framework that…

We propose an effective field-theoretic framework for analyzing Transformer attention through a thermodynamic lens. By constructing a Lagrangian on the information manifold equipped with the Fisher metric, we show that, within the…

Various forms of sparse attention have been explored to mitigate the quadratic computational and memory cost of the attention mechanism in transformers. We study sparse transformers not through a lens of efficiency but rather in terms of…

Pre-trained transformers exhibit the capability of adapting to new tasks through in-context learning (ICL), where they efficiently utilize a limited set of prompts without explicit model optimization. The canonical communication problem of…