Related papers: A completely uniform transformer for parity
Effectively processing long contexts is a critical challenge for language models. While standard Transformers are limited by quadratic complexity and poor length extrapolation, alternative architectures like sliding window attention and…
Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and heuristic subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes…
We propose a method for the construction of sets of variable dimension strong non-overlapping matrices basing on any strong non-overlapping set of strings.
The advent of Transformer-based models has surpassed the barriers of text. When working with speech, we must face a problem: the sequence length of an audio input is not suitable for the Transformer. To bypass this problem, a usual approach…
In odd dimensions the lattice overlap formalism is simpler than in even dimensions. Masslessness of fermions can still be preserved without fine tuning and gauge invariance without gauge averaging can be maintained, although, sometimes,…
The study on the expressive power of transformers shows that transformers are permutation equivariant, and they can approximate all permutation-equivariant continuous functions on a compact domain. However, these results are derived under…
Chain of thought is a natural inference-time method for increasing the computational power of transformer-based large language models (LLMs), but comes at the cost of sequential decoding. Are there more efficient alternatives to expand a…
We consider spatially coupled low-density parity-check codes with finite smoothing parameters. A finite smoothing parameter is important for designing practical codes that are decoded using low-complexity windowed decoders. By optimizing…
The tongue's intricate 3D structure, comprising localized functional units, plays a crucial role in the production of speech. When measured using tagged MRI, these functional units exhibit cohesive displacements and derived quantities that…
Transformer becomes the state-of-the-art translation model, while it is not well studied how each intermediate component contributes to the model performance, which poses significant challenges for designing optimal architectures. In this…
A novel method guaranteeing nondecreasing girth is presented for constructing longer low-density parity-check (LDPC) codes from shorter ones. The parity-check matrix of a shorter base code is decomposed into N (N>=2) non-overlapping…
We show that the Parikh image of the language of an NFA with n states over an alphabet of size k can be described as a finite union of linear sets with at most k generators and total size 2^{O(k^2 log n)}, i.e., polynomial for all fixed k…
Let $K$ denote a field and let $V$ denote a vector space over $K$ with finite positive dimension. We consider a pair of linear transformations $A : V \to V$ and $A^* : V \to V$ that satisfy (i) and (ii) below: (i) There exists a basis for…
We determine the generic consistency, dimension and nondegeneracy of the zero locus over $\mathbb{C}^*$, $\mathbb{R}^*$ and $\mathbb{R}_{>0}$ of vertically parametrized systems: parametric polynomial systems consisting of linear…
We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to…
Deep learning models have been widely applied in various aspects of daily life. Many variant models based on deep learning structures have achieved even better performances. Attention-based architectures have become almost ubiquitous in…
We present Layout Anything, a transformer-based framework for indoor layout estimation that adapts the OneFormer's universal segmentation architecture to geometric structure prediction. Our approach integrates OneFormer's task-conditioned…
We present numerical evidences using overlap fermions for a scale-invariant behavior of parity-invariant three-dimensional QED with two flavors of massless two-component fermions. Using finite-size scaling of the low-lying eigenvalues of…
Some Transformer-based models can perform cross-lingual transfer learning: those models can be trained on a specific task in one language and give relatively good results on the same task in another language, despite having been pre-trained…
Recent developments have shown the existence of quantum low-density parity check (qLDPC) codes with constant rate and linear distance. A natural question concerns the efficient decodability of these codes. In this paper, we present a linear…