Related papers: Theoretic Shaping Bounds for Single Letter Constra…
A new geometric shaping technique, referred to as coset shaping, is proposed and analyzed for coded QAM and PAM signaling. This method can be applied to both information and parity bits without introducing additional complexity. It is shown…
Two classes of turbo codes over high-order finite fields are introduced. The codes are derived from a particular protograph sub-ensemble of the (dv=2,dc=3) low-density parity-check code ensemble. A first construction is derived as a…
Decoding from large language models (LLMs) typically relies on fixed sampling hyperparameters (e.g., temperature, top-p), despite substantial variation in task difficulty and uncertainty across prompts and individual decoding steps. We…
The recent development of deep learning methods provides a new approach to optimize the belief propagation (BP) decoding of linear codes. However, the limitation of existing works is that the scale of neural networks increases rapidly with…
Speculative decoding is a powerful technique that attempts to circumvent the autoregressive constraint of modern Large Language Models (LLMs). The aim of speculative decoding techniques is to improve the average inference time of a large,…
Large language models (LLMs) for code rely on subword tokenizers, such as byte-pair encoding (BPE), learned from mixed natural language text and programming language code but driven by statistics rather than grammar. As a result,…
We derive a sphere-packing error exponent for coded transmission over discrete memoryless channels with a fixed decoding metric. By studying the error probability of the code over an auxiliary channel, we find a lower bound to the…
A new polar coding scheme for higher order modulation is presented. The proposed scheme is based on multi-level coding (MLC) with natural labeling, where the bit-level corresponding to the sign-bit is generated in dependence on the previous…
The performance of ``typical set (pairs) decoding'' for ensembles of Gallager's linear code is investigated using statistical physics. In this decoding, error happens when the information transmission is corrupted by an untypical noise or…
Recent systems on structured prediction focus on increasing the level of structural dependencies within the model. However, our study suggests that complex structures entail high overfitting risks. To control the structure-based…
Recently, a new class of codes, called sparse superposition or sparse regression codes, has been proposed for communication over the AWGN channel. It has been proven that they achieve capacity using power allocation and various forms of…
An additive noise channel is considered, in which the distribution of the noise is nonparametric and unknown. The problem of learning encoders and decoders based on noise samples is considered. For uncoded communication systems, the problem…
In source coding, either with or without side information at the decoder, the ultimate performance can be achieved by means of random binning. Structured binning into cosets of performing channel codes has been successfully employed in…
A ternary message passing (TMP) decoding algorithm for low-density parity-check codes is developed. All messages exchanged between variable and check nodes have a ternary alphabet, and the variable nodes exploit soft information from the…
Speculative decoding accelerates inference in large language models (LLMs) by generating multiple draft tokens simultaneously. However, existing methods often struggle with token misalignment between the training and decoding phases,…
This paper investigates the problem of variable-length lossy source coding allowing a positive excess distortion probability and an overflow probability of codeword lengths. Novel one-shot achievability and converse bounds of the optimal…
Despite the remarkable advances in language modeling, current mainstream decoding methods still struggle to generate texts that align with human texts across different aspects. In particular, sampling-based methods produce less-repetitive…
Speculative decoding accelerates large language model (LLM) inference by letting a lightweight draft model propose multiple tokens that the target model verifies in parallel. Yet existing training objectives optimize only a single greedy…
Decoding strategies play a central role in shaping the reasoning ability of large language models (LLMs). Traditional methods such as greedy decoding and beam search often suffer from error propagation, while sampling-based approaches…
Large language model (LLM) decoding involves generating a sequence of tokens based on a given context, where each token is predicted one at a time using the model's learned probabilities. The typical autoregressive decoding method requires…