Related papers: Expressivity of Transformers: A Tropical Geometry …
We reformulate the twistor construction for hyper- and quaternion-K\"ahler manifolds, introducing new sigma models that compute scalar potentials for the geometry. These sigma models have the twistor space of the quaternionic manifold as…
Recent advances in Transformer architectures have empowered their empirical success in a variety of tasks across different domains. However, existing works mainly focus on predictive accuracy and computational cost, without considering…
We show that a constant number of self-attention layers can efficiently simulate, and be simulated by, a constant number of communication rounds of Massively Parallel Computation. As a consequence, we show that logarithmic depth is…
Is it always necessary to compute tokens from shallow to deep layers in Transformers? The continued success of vanilla Transformers and their variants suggests an undoubted "yes". In this work, however, we attempt to break the depth-ordered…
In this work, we study how multi-head latent attention (MLA), a popular strategy for compressing key/value memory, affects a transformer's internal capacity during pretraining. Using a lightweight suite of Marchenko-Pastur (MP) diagnostics,…
We propose Axial Transformers, a self-attention-based autoregressive model for images and other data organized as high dimensional tensors. Existing autoregressive models either suffer from excessively large computational resource…
Social group activity recognition is a challenging task extended from group activity recognition, where social groups must be recognized with their activities and group members. Existing methods tackle this task by leveraging region…
We develop a quenched thermodynamic formalism for open random dynamical systems generated by finitely branched, piecewise-monotone mappings of the interval. The openness refers to the presence of holes in the interval, which terminate…
A geometric theory of the irreducible tensor operators of quantum spin systems. It is based upon the Maxwell-Sylvester geometric representation of the multipolar electrostatic potential. In the latter, an order-$\ell$ multipolar potential…
Transformer model architectures have become an indispensable staple in deep learning lately for their effectiveness across a range of tasks. Recently, a surge of "X-former" models have been proposed which improve upon the original…
This paper explores the electrical and thermal conductivity of complex contact spots on the surface of a half-space. Employing an in-house Fast Boundary Element Method implementation, various complex geometries were studied. Our…
Compressible multiphase and multicomponent solvers require accurate interface representation without spurious pressure oscillations. At material interfaces, pressure and velocity are continuous while density and the equation of state…
The Mechanistic Interpretability (MI) program has mapped the Transformer as a precise computational graph. We extend this graph with a conservation law and time-varying AC dynamics, viewing it as a physical circuit. We introduce Momentum…
We investigate the realizability of balanced functions on tropical curves, establishing new sufficient criteria for superabundant functions on genus two curves, analogous to the well-spacedness condition in genus one. We find that…
Transient gas network simulations can significantly assist in design and operational aspects of gas networks. Models used in these simulations require a detailed framework integrating various models of the network constituents - pipes and…
We developed the full magnetohydrodynamical analytical jet model that allows accurate reproducing of a transversal and longitudinal structure for a highly collimated relativistic jets. This model can be used as a setup for convenient…
We review connections between phase transitions in high-dimensional combinatorial geometry and phase transitions occurring in modern high-dimensional data analysis and signal processing. In data analysis, such transitions arise as abrupt…
Transformers have achieved remarkable successes across a wide range of applications, yet the theoretical foundation of their model efficiency remains underexplored. In this work, we investigate how the model parameters -- mainly attention…
A quasi-geostrophic intermediate complexity model is considered, providing a schematic representation of the baroclinic conversion processes which characterize the physics of the mid-latitudes atmospheric circulation. The model is relaxed…
We show that the tropicalization of an irreducible d-dimensional variety over a field of characteristic 0 is (d-l)-connected through codimension one, where l is the dimension of the lineality space of the tropicalization. From this we…