Related papers: Theoretic Shaping Bounds for Single Letter Constra…
We present a family of algorithms, combining real-space renormalization methods and belief propagation, to estimate the free energy of a topologically ordered system in the presence of defects. Such an algorithm is needed to preserve the…
Language models (LMs) are trained on billions of tokens in an attempt to recover the true language distribution. Still, vanilla random sampling from LMs yields low quality generations. Decoding algorithms attempt to restrict the LM…
Traditional communication theory focuses on minimizing transmit power. However, communication links are increasingly operating at shorter ranges where transmit power can be significantly smaller than the power consumed in decoding. This…
This paper considers the problem of channel coding over Gaussian intersymbol interference (ISI) channels with a given metric decoding rule. Specifically, it is assumed that the mismatched decoder has an incorrect assumption on the impulse…
Token-based masked generative models are gaining popularity for their fast inference time with parallel decoding. While recent token-based approaches achieve competitive performance to diffusion-based models, their generation performance is…
The problem of compression in standard information theory consists of assigning codes as short as possible to numbers. Here we consider the problem of optimal coding -- under an arbitrary coding scheme -- and show that it predicts Zipf's…
Throughput and per-packet delay can present strong trade-offs that are important in the cases of delay sensitive applications.We investigate such trade-offs using a random linear network coding scheme for one or more receivers in single hop…
In this paper we investigate the \emph{approximate string matching problem} when the allowed edit operations are \emph{non-overlapping unbalanced translocations of adjacent factors}. Such kind of edit operations take place when two adjacent…
Probabilistic amplitude shaping (PAS) is on track to become the de facto coded modulation standard for communication systems aiming to operate close to channel capacity at high transmission rates. The essential component of PAS that breeds…
Matched decoding is a technique that enables the efficient maximum-likelihood sequence estimation of convolutionally encoded PAM-transmission over ISI-channels. Recently, we have shown that the super-trellis of encoder and channel can be…
For finite coupling lengths, terminated spatially coupled low-density parity-check (SC-LDPC) codes show a non-negligible rate-loss. In this paper, we investigate if this rate loss can be mitigated by tail-biting SC-LDPC codes in conjunction…
Speculative generation has emerged as a promising technique to accelerate inference in large language models (LLMs) by leveraging parallelism to verify multiple draft tokens simultaneously. However, the fundamental limits on the achievable…
This paper is focused on the performance analysis of binary linear block codes (or ensembles) whose transmission takes place over independent and memoryless parallel channels. New upper bounds on the maximum-likelihood (ML) decoding error…
We analyze random coding error exponents associated with erasure/list Slepian-Wolf decoding using two different methods and then compare the resulting bounds. The first method follows the well known techniques of Gallager and Forney and the…
Abstract: In this article, we will analyze in detail the coding limit of an individual sequence by introducing the latest developments brought by the Set Shaping Theory. This new theory made us realize that there is a huge difference…
A missing piece in quantum information theory, with very few exceptions, has been to provide the random coding exponents for quantum information-processing protocols. We remedy the situation by providing these exponents for a variety of…
With the widespread application of Large Language Models (LLMs) in the field of Natural Language Processing (NLP), enhancing their performance has become a research hotspot. This paper presents a novel multi-prompt ensemble decoding…
Speculative decoding has emerged as a pivotal technique to accelerate LLM inference by employing a lightweight draft model to generate candidate tokens that are subsequently verified by the target model in parallel. However, while this…
Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models (LLMs) by employing a small language model to draft a hypothesis sequence, which is then validated by the LLM. The effectiveness…
In distributed optimization problems, a technique called gradient coding, which involves replicating data points, has been used to mitigate the effect of straggling machines. Recent work has studied approximate gradient coding, which…