Related papers: Expressivity of Transformers: A Tropical Geometry …
This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into sequence context.…
For tropical $n$-variable polynomials $f, g$ a criterion of containment for tropical hypersurfaces $Trop(f)\subset Trop(g)$ is provided in terms of their Newton polyhedra $N(f), N(g)\subset \mathbb{R}^{n+1}$. Namely, $Trop(f)\subset…
Using tropical geometry one can translate problems in enumerative geometry to combinatorial problems. Thus tropical geometry is a powerful tool in enumerative geometry over the complex and real numbers. Results from $\mathbb{A}^1$-homotopy…
We study the capacity of the self-attention key-query channel: for a fixed budget, how many distinct token-token relations can a single layer reliably encode? We introduce Relational Graph Recognition, where the key-query channel encodes a…
Expressivity plays a fundamental role in evaluating deep neural networks, and it is closely related to understanding the limit of performance improvement. In this paper, we propose a three-pipeline training framework based on critical…
In tropical geometry, one studies algebraic curves using combinatorial techniques via the tropicalization procedure. The tropicalization depends on a map to an algebraic torus and the combinatorial methods are most useful when the…
We study the behavior of phylogenetic tree shapes in the tropical geometric interpretation of tree space. Tree shapes are formally referred to as tree topologies; a tree topology can also be thought of as a tree combinatorial type, which is…
We present TropNNC, a framework for compressing neural networks with linear and convolutional layers and ReLU activations using tropical geometry. By representing a network's output as a tropical rational function, TropNNC enables…
The Transformer architecture has achieved tremendous success in natural language processing, computer vision, and scientific computing through its self-attention mechanism. However, its core components-positional encoding and attention…
Compressibility transformations are used to relate hypersonic zero-pressure-gradient (ZPG) turbulent boundary layers (TBLs) to incompressible reference states, but their assessment has largely focused on the collapse of transformed mean…
Transformers serve as the foundation of most modern large language models. To mitigate the quadratic complexity of standard full attention, various efficient attention mechanisms, such as linear and hybrid attention, have been developed. A…
Measure-theoretic slow entropy is a more refined invariant than the classical measure-theoretic entropy to characterize the complexity of dynamical systems with subexponential growth rates of distinguishable orbit types. In this paper we…
In deep learning theory, the covariance matrix of the representations serves as a proxy to examine the network's trainability. Motivated by the success of Transformers, we study the covariance matrix of a modified Softmax-based attention…
We consider the response of a multicomponent body to $n$ fields, such as electric fields, magnetic fields, temperature gradients, concentration gradients, etc., where each component, which is possibly anisotropic, may cross couple the…
The transformer has revolutionized modern AI across language, vision, and beyond. It consists of $L$ layers, each running $H$ attention heads in parallel and feeding the combined output to the subsequent layer. In attention, the input…
Under real-analytic assumptions on decoder-only Transformers, recent work shows that the map from discrete prompts to last-token hidden states is generically injective on finite prompt sets. We refine this picture: for each layer $\ell$ we…
We study algebraic and combinatorial aspects of (classical) projections of $m$-dimensional tropical varieties onto $(m+1)$-dimensional planes. Building upon the work of Sturmfels, Tevelev, and Yu on tropical elimination as well as the work…
We study the geometry of tropical extensions of hyperfields, including the ordinary, signed and complex tropical hyperfields. We introduce the framework of 'enriched valuations' as hyperfield homomorphisms to tropical extensions, and show…
Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the attention function, which learns dependencies between tokens and…
The transformer is the most popular neural architecture for language modeling. The cornerstone of the transformer is its global attention mechanism, which lets the model aggregate information from all preceding tokens before generating the…