Related papers: Expressivity of Transformers: A Tropical Geometry …
This paper presents a unified mathematical framework for inference in graphical models, building on the observation that graphical models are algebraic varieties. From this geometric viewpoint, observations generated from a model are…
Transformers evaluated in a single, fixed-depth pass are provably limited in expressive power to the constant-depth circuit class TC0. Running a Transformer autoregressively removes that ceiling -- first in next-token prediction and, more…
We introduce the tropical $F$-polynomial $f_M$ of a quiver representation $M$. We study its interplay with the general presentation for any finite-dimensional basic algebra. We give an interpretation of evaluating $f_M$ at a weight vector.…
Using the interpretation of the ultradiscretization procedure as a non-Archimedean valuation, we use results of tropical geometry to show how roots and poles manifest themselves in piece-wise linear systems as points of…
Transformers have achieved extraordinary success in modern machine learning due to their excellent ability to handle sequential data, especially in next-token prediction (NTP) tasks. However, the theoretical understanding of their…
Speyer recognized that matroids encode the same data as a special class of tropical linear spaces and Shaw interpreted tropically certain basic matroid constructions; additionally, Frenk developed the perspective of tropical linear spaces…
We present a high-accuracy procedure for electronic structure calculations of strongly correlated materials. To address limitations in current electronic structure methods, we employ density functional theory in combination with the…
Tensor Attention extends traditional attention mechanisms by capturing high-order correlations across multiple modalities, addressing the limitations of classical matrix-based attention. Meanwhile, Rotary Position Embedding…
Configurational polyhedora (CP) is a hyperpolyhedra on multidimensional configuration space, whose vertex (and edges) corresponds to upper or lower limit value of correlation functions for all possible atomic configuration on given lattice.…
The purpose of this paper is fourfold. The first is to develop the theory of tropical differential algebraic geometry from scratch; the second is to present the tropical fundamental theorem for differential algebraic geometry, and show how…
We establish existence of global-in-time weak solutions to the one dimensional, compressible Navier-Stokes system for a viscous and heat conducting ideal polytropic gas (pressure $p=K\theta/\tau$, internal energy $e=c_v \theta$), when the…
Attention layers, as commonly used in transformers, form the backbone of modern deep learning, yet there is no mathematical description of their benefits and deficiencies as compared with other architectures. In this work we establish both…
Despite the popularity of transformers in practice, their architectures are empirically designed and neither mathematically justified nor interpretable. Moreover, as indicated by many empirical studies, some components of transformer…
We derive a graph expansion for the thermal partition function of solvable two-dimensional models with boundaries. This expansion of the integration measure over the virtual particles winding around the time cycle is obtained with the help…
Self-attention in Transformers is typically implemented as $\mathrm{softmax}(QK^\top/\sqrt{d})V$, where $Q=XW_Q$, $K=XW_K$, and $V=XW_V$ are learned linear projections of the input $X$. We ask whether these learned projections are…
We undertake a combinatorial study of the piecewise linear map g : R^{2m+2n} --> R^{mn} which assigns to the four vectors a, A in R^m and b, B in R^n the m by n matrix given by g_{ij} = min (a_i + b_j, A_i+B_j). This map arises naturally in…
In this paper, we aim to build the global convergence theory of encoder-only shallow Transformers under a realistic setting from the perspective of architectures, initialization, and scaling under a finite width regime. The difficulty lies…
We quantify the topological expansion properties of bounded degree simplicial complexes in terms of a family of sublinear functions, in analogy with the separation profile of Benjamini-Schramm-Tim\'ar for classical expansion of bounded…
A tropical expansion is a degeneration of a toroidal embedding, induced by a polyhedral subdivision of its tropicalisation. Each irreducible component of a tropical expansion admits a collapsing map down to a stratum of the original…
We propose to study the tropical geometry specifically arising from convergent Hahn series in multiple indeterminates. One application is a new view on stable intersections of tropical hypersurfaces. Another one is perturbations of rank one…