Related papers: Scaling Limits of Long-Context Transformers
The softmax function is crucial in Transformer attention, which normalizes each row of the attention scores with summation to one, achieving superior performances over other alternative functions. However, the softmax function can face a…
The ability to process long contexts is crucial for many natural language processing tasks, yet it remains a significant challenge. While substantial progress has been made in enhancing the efficiency of attention mechanisms, there is still…
There are two independent critical exponents that describe the behavior of systems near their critical point. However, at the critical point only the exponent $\eta$, which describes the decay of the correlation function, is usually…
The jamming transition of particles with finite-range interactions is characterized by a variety of critical phenomena, including power law distributions of marginal contacts. We numerically study a recently proposed simple model of…
We examine a phase transition in a model of random spatial permutations which originates in a study of the interacting Bose gas. Permutations are weighted according to point positions; the low-temperature onset of the appearance of…
Transformers have become the go-to architecture for language and vision tasks, yet their theoretical properties, especially memorization capacity, remain elusive. This paper investigates the memorization abilities of multi-head attention…
We simulated site dilute Ising models in $d=3$ dimensions for several lattice sizes $L$. For each $L$ singular thermodynamic quantities $X$ were measured at criticality and their distributions $P(X)$ were determined, for ensembles of…
Softmax Self-Attention (SSA) is a key component of Transformer architectures. However, when utilised within skipless architectures, which aim to improve representation learning, recent work has highlighted the inherent instability of SSA…
Let $\Lambda=\{\Lambda_0,\Lambda_1,\Lambda_2,\ldots\}$ be the point process that describes the edge scaling limit of either (i) "regular" beta-ensembles with inverse temperature $\beta>0$, or (ii) the top eigenvalues of Wishart or Gaussian…
We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specific content of the context and depends only on its length. We…
Modern sequence modeling is dominated by two families: Transformers, whose self-attention can access arbitrary elements of the visible sequence, and structured state-space models, which propagate information through an explicit recurrent…
The Ising model in two dimensions with the special boundary conditions of Brascamp and Kunz is analysed. Leading and sub-dominant scaling behaviour of the Fisher zeroes are determined exactly. The finite-size scaling, with corrections, of…
Extreme events can come either from point processes, when the size or energy of the events is above a certain threshold, or from time series, when the intensity of a signal surpasses a threshold value. We are particularly concerned by the…
We analyze the block averaging transformation applied to lattice gas models with short range interaction in the uniqueness region below the critical temperature. We prove weak Gibbsianity of the renormalized measure and convergence of the…
Attention mechanism is a central component of the transformer architecture which led to the phenomenal success of large language models. However, the theoretical principles underlying the attention mechanism are poorly understood,…
Extensive efforts have been made to boost the performance in the domain of language models by introducing various attention-based transformers. However, the inclusion of linear layers with large dimensions contributes to significant…
We study the dynamical response of a system to a sudden change of the tuning parameter $\lambda$ starting (or ending) at the quantum critical point. In particular we analyze the scaling of the excitation probability, number of excited…
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks. However, the quadratic complexity of softmax attention remains a central bottleneck that limits their scalability. Alman and Song (NeurIPS…
A $d$--dimensional quantum model in the spherical approximation confined to a general geometry of the form $L^{d-d^{\prime}} \times\infty^{d^{\prime}}\times L_{\tau}^{z}$ ($L$--linear space size and $L_{\tau}$--temporal size) and subjected…
Linear attention reduces the quadratic cost of softmax attention to $\mathcal{O}(T)$, but its memory state grows as $\mathcal{O}(T)$ in Frobenius norm, causing progressive interference between stored associations. We introduce…