Related papers: A completely uniform transformer for parity
We believe three ingredients are needed for further progress in persistence and its use: invariants not relying on decomposition theorems to go beyond 1-dimension, outcomes suitable for statistical analysis and a setup adopted for…
A constant-dimension code (CDC) is a set of subspaces of constant dimension in a common vector space with upper bounded pairwise intersection. We improve and generalize two constructions for CDCs, the improved linkage construction and the…
In this paper we propose two variants of the substructuring preconditioner for solving three-dimensional elliptic-type equations with strongly discontinuous coefficients. In the new preconditioners, we use the simplest coarse solver…
This paper introduces a novel modification of the transformer architecture, tailored for the data-efficient pretraining of language models. This aspect is evaluated by participating in the BabyLM challenge, where our solution won both the…
We study universal aspects of polymer conformations and transverse fluctuations for a single swollen chain characterized by a contour length $L$ and a persistence length $\ell_p$ in two dimensions (2D) and in three dimensions (3D) in the…
The decoder-only Transformer architecture with causal masking and relative position encoding (RPE) has become the de facto choice in language modeling. Despite its exceptional performance across various tasks, we have identified two…
We show that every flat nonlinear discrete-time system with two inputs can be transformed into a structurally flat normal form by state- and input transformations. This normal form has a triangular structure and allows to read off the flat…
Transformer has been widely-used in many Natural Language Processing (NLP) tasks and the scaled dot-product attention between tokens is a core module of Transformer. This attention is a token-wise design and its complexity is quadratic to…
Positional encodings are a core part of transformer-based models, enabling processing of sequential data without recurrence. This paper presents a theoretical framework to analyze how various positional encoding methods, including…
This paper investigates singular configurations of planar 3-RPR parallel manipulators, which result from applying the averaging technique to solution pairs of their direct kinematic problem. Without computing the zeros of the corresponding…
After their initial success in natural language processing, transformer architectures have rapidly gained traction in computer vision, providing state-of-the-art results for tasks such as image classification, detection, segmentation, and…
The parametrization theorem is derived in a flat nD pseudo-complex affine space. The pseudo-complex hyperbolic space accomodates n-number of uncompactified time-like extra dimensions with sugnature (s,r), where s and r are the numbers of…
Parity functions are fundamental Boolean operations with critical applications across machine learning, cryptography, and error correction. Yet, learning high-dimensional parity functions poses significant challenges: in a general setting,…
In this paper we construct nonlinear partial differential equations in more than 3 independent variables, possessing a manifold of analytic solutions with high, but not full, dimensionality. For this reason we call them ``partially…
High-performance machine learning tools in particle physics rest on two complementary directions: encoding symmetries explicitly in the architecture, and implicitly learning the structure of the data through large-scale (pre-) training. We…
Multimodal large language models (MLLMs) have made significant progress in vision-language understanding, yet effectively aligning different modalities remains a fundamental challenge. We present a framework that unifies multimodal…
We show that a constant number of self-attention layers can efficiently simulate, and be simulated by, a constant number of communication rounds of Massively Parallel Computation. As a consequence, we show that logarithmic depth is…
For any translation-invariant quantum lattice system with a symmetry group G, we propose a practical and universal construction of order parameters which identify quantum phase transitions with symmetry-breaking order. They are defined in…
We numerically determine subleading scaling terms in the ground-state entanglement entropy of several two-dimensional (2D) gapless systems, including a Heisenberg model with N\'eel order, a free Dirac fermion in the {\pi}-flux phase, and…
The availability of corpora to train semantic parsers in English has lead to significant advances in the field. Unfortunately, for languages other than English, annotation is scarce and so are developed parsers. We then ask: could a parser…