Related papers: Expressivity of Transformers: A Tropical Geometry …
Can algebraic geometry enhance the sharpness, robustness, and interpretability of modern neural reasoning models by equipping them with a mathematically grounded inductive bias? To answer this, we introduce Tropical Attention, an attention…
We prove that the Transformer self-attention mechanism in the high-confidence regime ($\beta \to \infty$, where $\beta$ is an inverse temperature) operates in the tropical semiring (max-plus algebra). In particular, we show that taking the…
We propose an algebraic geometric framework to study the expressivity of linear activation neural networks. A particular quantity of neural networks that has been actively studied is the number of linear regions, which gives a…
In this paper we use the connections between tropical algebraic geometry and rigid analytic geometry in order to prove two main results. We use tropical methods to prove a theorem about the Newton polygon for convergent power series in…
Suppose that there exists a hypersurface with the Newton polytope $\Delta$, which passes through a given set of subvarieties. Using tropical geometry, we associate a subset of $\Delta$ to each of these subvarieties. We prove that a weighted…
We study how geometric properties of tropical convex sets and polytopes, which are of interest in many application areas, manifest themselves in their algebraic structure as modules over the tropical semiring. Our main results establish a…
The motivic nearby fiber is an invariant obtained from degenerating a complex variety over a disc. It specializes to the Euler characteristic of the original variety but also contains information on the variation of Hodge structure…
We find a relation between mixed volumes of several polytopes and the convex hull of their union, deducing it from the following fact: the mixed volume of a collection of polytopes only depends on the product of their support functions…
Hyperbolic geometry have shown significant potential in modeling complex structured data, particularly those with underlying tree-like and hierarchical structures. Despite the impressive performance of various hyperbolic neural networks…
Despite the remarkable empirical performance of Transformers, their theoretical understanding remains elusive. Here, we consider a deep multi-head self-attention network, that is closely related to Transformers yet analytically tractable.…
In tolerancing analysis, geometrical or contact specifications can be represented by polytopes. Due to the degrees of invariance of surfaces and that of freedom of joints, these operand polytopes are originally unbounded in most of the…
We examine the intrinsic (within the attention head) and extrinsic (amongst the attention heads) structure of the self-attention mechanism in transformers. Theoretical evidence for invariance of the self-attention mechanism to softmax…
We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the…
Tropical refined invariants of toric surfaces constitute a fascinating interpolation between real and complex enumerative geometries via tropical geometry. They were originally introduced by Block and G\"ottsche, and further extended by…
We revisit the Universal Approximation Theorem(UAT) through the lens of the tropical geometry of neural networks and introduce a constructive, geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs). Tropical geometry…
This paper is about the combinatorics of finite point configurations in the tropical projective space or, dually, of arrangements of finitely many tropical hyperplanes. Moreover, arrangements of finitely many tropical halfspaces can be…
Recently, concepts from the emerging field of tropical geometry have been used to identify different scaling regimes in chemical reaction networks where dimension reduction may take place. In this paper, we try to formalize these ideas…
We develop a combinatorial framework to study certain polyhedral maps which are higher-dimensional analogues of tropical covers between metric graphs. Under a mild combinatorial assumption, we show that a map satisfies the so-called…
We investigate a canonical extension of a conventional combinatorial notion of reduced divisors to a notion of tropical projections, which can be defined as the unique minimizers of the so-called $B$-pseudonorms with respect to compact…
The vanilla self-attention mechanism in Transformers can be viewed as a two-layer fast-weight MLP, whose weights are dynamically induced by inputs and whose hidden dimension is equal to the sequence length $N$. As the context extends, the…