Related papers: Scaling Limits of Long-Context Transformers

Transformer has become the dominant architecture for sequence modeling, yet a detailed understanding of how its structural parameters influence expressive power remains limited. In this work, we study the approximation properties of…

Transformers are increasingly adopted for modeling and forecasting time-series, yet their internal mechanisms remain poorly understood from a dynamical systems perspective. In contrast to classical autoregressive and state-space models,…

We investigate the low-temperature critical behavior of the three dimensional random-field Ising ferromagnet. By a scaling analysis we find that in the limit of temperature $T \to 0$ the usual scaling relations have to be modified as far as…

We study the convergence and shape correction to the limit distributions of extreme values due to the finite size (FS) of data sets. A renormalization method is introduced for the case of independent, identically distributed (iid)…

Recently a number of empirical "universal" scaling law papers have been published, most notably by OpenAI. `Scaling laws' refers to power-law decreases of training or test error w.r.t. more data, larger neural networks, and/or more compute.…

Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity and linearly increased cache size. Sliding window attention…

Despite the progress made in sentence-level NMT, current systems still fall short at achieving fluent, good quality translation for a full document. Recent works in context-aware NMT consider only a few previous sentences as context and may…

Exact results of the finite-size behavior of the susceptibility in three-dimensional mean spherical model films under Dirichlet-Dirichlet, Dirichlet-Neumann and Neumann-Neumann boundary conditions are presented. The corresponding scaling…

Despite powering modern AI, transformers remain mysteriously brittle to train. We develop a stability theory that explains why pre-LayerNorm works, why DeepNorm uses $N^{-1/4}$ scaling, and why warmup is necessary, all from first…

Attention is the critical component of a transformer. Yet the quadratic computational complexity of vanilla full attention in the input size and the inability of its linear attention variant to focus have been challenges for computer vision…

We revisit the critical behavior of the sub-ohmic spin-boson model. Analysis of both the leading and subleading terms in the temperature dependence of the inverse static local spin susceptibility at the quantum critical point, calculated…

In this article, we revisit the question of fluctuations of linear statistics of beta ensembles in the single cut and non-critical regime for general potentials $V$ under mild regularity and growth assumptions. Our main objective is to…

We give evidence of a clear structural signature of the glass transition, in terms of a static correlation length with the same dependence on the system size which is typical of critical phenomena. Our approach is to introduce an external,…

The time evolution of systems relaxing towards thermal equilibrium is examined near the critical temperature $T_c$, with special attention paid to the role of the initial value $m_i$ of the order parameter $\phi$. To this end, the…

Sequence models face a fundamental tradeoff between memory capacity and computational efficiency. Transformers achieve expressive context modeling at quadratic cost, while linear attention and state-space models run in linear time by…

In the classical $\beta$-ensembles of random matrix theory, setting $\beta = 2 \alpha/N$ and taking the $N \to \infty$ limit gives a statistical state depending on $\alpha$. Using the loop equations for the classical $\beta$-ensembles, we…

Recent experiments and computer simulations show that supercooled liquids around the glass transition temperature are "dynamically heterogeneous" [1]. Such heterogeneity is expected from the random first order transition theory of the glass…

The interplay between disorder, quantum fluctuations and dissipation is studied in the random transverse Ising chain coupled to a dissipative Ohmic bath with a real space renormalization group. A typically very large length scale, L*, is…

We study equilibrium configurations of non-Euclidean plates, in which the reference metric is uniaxially periodic. This work is motivated by recent experiments on thin sheets of composite thermally responsive gels [1]. Such sheets bend…

Large Language Models (LLMs) exhibit exceptional proficiency in handling extensive context windows in natural language. Nevertheless, the quadratic scaling of attention computation relative to sequence length creates substantial efficiency…