Related papers: A completely uniform transformer for parity

The surface code is a two-dimensional topological code with code parameters that scale optimally with the number of physical qubits, under the constraint of two-dimensional locality. In three spatial dimensions an analogous simple yet…

This paper consists of three parts. The first part presents a large class of new binary quasi-cyclic (QC)-LDPC codes with girth of at least 6 whose parity-check matrices are constructed based on cyclic subgroups of finite fields.…

Recent advancements in transformer-based models have greatly improved time series analysis, providing robust solutions for tasks such as forecasting, anomaly detection, and classification. A crucial element of these models is positional…

We introduce a new way of learning to encode position information for non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which contain inductive bias by loading the input tokens sequentially, non-recurrent models are…

Multilingual Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear. Do they form shared multilingual representations with language-specific decoding, and if so, why does…

We propose a generalized form of entangled coherent states (ECS) and apply them in a multi-arm optical interferometer to estimate multiple phase shifts. We obtain the quantum Cramer-Rao bounds for both the linear and nonlinear…

Large language models are capable of in-context learning, the ability to perform new tasks at test time using a handful of input-output examples, without parameter updates. We develop a universal approximation theory to elucidate how…

We discuss a systematic construction of dimensionless quantum-mechanical equations. The process reduces the number of independent model parameters to a minimum and, at the same time, provides the natural units of length, energy, etc. in a…

Although the Transformer translation model (Vaswani et al., 2017) has achieved state-of-the-art performance in a variety of translation tasks, how to use document-level context to deal with discourse phenomena problematic for Transformer…

In this paper, a construction of $(n,k,\delta)$ LDPC convolutional codes over arbitrary finite fields, which generalizes the work of Robinson and Bernstein and the later work of Tong is provided. The sets of integers forming a…

Generalizing to longer sentences is important for recent Transformer-based language models. Besides algorithms manipulating explicit position features, the success of Transformers without position encodings (NoPE) provides a new way to…

Spatially-coupled low-density parity-check codes attract much attention due to their capacity-achieving performance and a memory-efficient sliding-window decoding algorithm. On the other hand, the encoder needs to solve large linear…

We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their…

The concept of universality has shaped our understanding of many-body physics, but is mostly limited to homogenous systems. Here, we present a study of universality on a non-homogeneous graph, the long-range diluted graph (LRDG). Its…

Transformers have supplanted recurrent models in a large number of NLP tasks. However, the differences in their abilities to model different syntactic properties remain largely unknown. Past works suggest that LSTMs generalize very well on…

We show that the equivalence of deterministic linear top-down tree-to-word transducers is decidable in polynomial time. Linear tree-to-word transducers are non-copying but not necessarily order-preserving and can be used to express XML and…

We study the r-th elementary symmetric polynomial in $n$ variables with 2<r<n. There are two kinds of linear transformations on the parameter space that leave this polynomial invariant: Namely, any permutation of the variables and…

Current methods for image-to-image translation produce compelling results, however, the applied transformation is difficult to control, since existing mechanisms are often limited and non-intuitive. We propose ParGAN, a generalization of…

Unequal transition probabilities between input and output symbols, input power constraints, or input symbols of unequal durations can lead to non-uniform capacity achieving input distributions for communication channels. Using uniform input…

In this paper we present a Transformer-Transducer model architecture and a training technique to unify streaming and non-streaming speech recognition models into one model. The model is composed of a stack of transformer layers for audio…