Related papers: A completely uniform transformer for parity

Understanding where transformer language models encode psychologically meaningful aspects of meaning is essential for both theory and practice. We conduct a systematic layer-wise probing study of 58 psycholinguistic features across 10…

Priority encoders are typically considered expensive hardware components in terms of complexity, especially at high bit precisions or input lengths (e.g., above 512 bits). However, if the complexity can be reduced, priority encoders can…

In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTMs, CPT is designed to utilize the shared knowledge between…

We construct an example of an iterated function system on the line, consisting of linear fractional transformations, such that two of the maps share a fixed points, but the dimension of the attractor equals the conformal dimension, so that…

Codes considered as structures within unit schemes greatly extends the availability of linear block and convolutional codes and allows the construction of these codes to required length, rate, distance and type. Properties of a code emanate…

We investigate multi-scale transformer language models that learn representations of text at multiple scales, and present three different architectures that have an inductive bias to handle the hierarchical nature of language. Experiments…

This work builds together two popular blocks of neural architecture, namely convolutional layers and Transformers, for large language models (LLMs). Non-causal conformers are used ubiquitously in automatic speech recognition. This work aims…

The notion of entangling power of unitary matrices was introduced by Zanardi, Zalka and Faoro [PRA, 62, 030301]. We study the entangling power of permutations, given in terms of a combinatorial formula. We show that the permutation matrices…

This paper presents a novel unifying framework of bilinear LSTMs that can represent and utilize the nonlinear interaction of the input features present in sequence datasets for achieving superior performance over a linear LSTM and yet not…

Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto…

While a considerable amount of semantic parsing approaches have employed RNN architectures for code generation tasks, there have been only few attempts to investigate the applicability of Transformers for this task. Including hierarchical…

We investigate how embedding dimension affects the emergence of an internal "world model" in a transformer trained with reinforcement learning to perform bubble-sort-style adjacent swaps. Models achieve high accuracy even with very small…

Channel coding is vital for reliable sixth-generation (6G) data transmission, employing diverse error correction codes for various application scenarios. Traditional decoders require dedicated hardware for each code, leading to high…

Multilingual Machine Translation promises to improve translation quality between non-English languages. This is advantageous for several reasons, namely lower latency (no need to translate twice), and reduced error cascades (e.g., avoiding…

In NLP, a large volume of tasks involve pairwise comparison between two sequences (e.g. sentence similarity and paraphrase identification). Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders.…

Scaling laws for large language models (LLMs) have provided useful guidance in training ever larger models for predictable performance gains. Time series forecasting shares a similar sequential structure to language, and is amenable to…

In this paper, we propose a comprehensive Polar coding solution that integrates reliability calculation, rate matching and parity-check coding. Judging a channel coding design from the industry's viewpoint, there are two primary concerns:…

In this paper, the concept of the {\it broken diagonal pair} in the chess-like square board is used to define some well-structured block designs whose incidence matrices can be considered as the parity-check matrices of some high rate cycle…

While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision. Existing equivariant methods often sacrifice the efficiency and flexibility that make Transformers so effective through…

In this paper, we obtain the classification theorem for three-dimensional complete space-like $\lambda$-translators $x:M^{3} \rightarrow \mathbb R^{4}_{1}$ with constant norm of the second fundamental form and constant $f_{4}$ in the…