相关论文: Maxwell Construction: The Hidden Bridge between It…

Transformers process tokens in parallel but are temporally shallow: at position $t$, each layer attends to key-value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrent models offer…

We consider recursive decoding techniques for RM codes, their subcodes, and newly designed codes. For moderate lengths up to 512, we obtain near-optimum decoding with feasible complexity.

K.A.S. Immink and J.H. Weber recently defined and studied a channel with both gain and offset mismatch, modelling the behaviour of charge-leakage in flash memory. They proposed a decoding measure for this channel based on minimising Pearson…

In the quest for scalable Bayesian computational algorithms we need to exploit the full potential of existing methodologies. In this note we point out that message passing algorithms, which are very well developed for inference in graphical…

We consider transmission of two independent and separately encoded sources over a two-user binary-input Gaussian multiple-access channel. The channel gains are assumed to be unknown at the transmitter and the goal is to design an…

We consider the N-user broadcast erasure channel with public feedback and side information. Before the beginning of transmission, each receiver knows a function of the messages of some of the other receivers. This situation arises naturally…

We consider the hashing mechanism for constructing binary embeddings, that involves pseudo-random projections followed by nonlinear (sign function) mappings. The pseudo-random projection is described by a matrix, where not all entries are…

While many recent methods aim to unlearn or remove knowledge from pretrained models, seemingly erased knowledge often persists and can be recovered in various ways. Because large foundation models are far from interpretable, understanding…

In the past decade several theoretical Maxwell's demon models have been proposed exhibiting effects such as refrigerating, doing work at the cost of information, and some experiments have been done to realise these effects. Here we propose…

For information transmission a binary symmetric channel is used. There is also another noisy binary symmetric channel (feedback channel), and the transmitter observes without delay all the outputs of the forward channel via that feedback…

Shannon's channel coding theorem characterizes the maximal rate of information that can be reliably transmitted over a communication channel when optimal encoding and decoding strategies are used. In many scenarios, however, practical…

Motivation: In the last few years a growing interest in biology has been shifting towards the problem of optimal information extraction from the huge amount of data generated via large scale and high-throughput techniques. One of the most…

Traditional communication theory focuses on minimizing transmit power. However, communication links are increasingly operating at shorter ranges where transmit power can be significantly smaller than the power consumed in decoding. This…

Information embedding (IE) is the transmission of information within a host signal subject to a distortion constraint. There are two types of embedding methods, namely irreversible IE and reversible IE, depending upon whether or not the…

Using projection between Euclidian spaces of different dimensions, the signal compression and decompression become straightforward. This encoding/decoding technique requires no preassigned measuring matrix as in compressed sensing.…

Understanding the functional principles of information processing in deep neural networks continues to be a challenge, in particular for networks with trained and thus non-random weights. To address this issue, we study the mapping between…

We study how information propagates in decoder-only Transformers, which are the architectural backbone of most existing frontier large language models (LLMs). We rely on a theoretical signal propagation analysis -- specifically, we analyse…

The code that combines channel estimation and error protection has received general attention recently, and has been considered a promising methodology to compensate multi-path fading effect. It has been shown by simulations that such code…

We develop a novel framework for proving converse theorems for channel coding, which is based on the analysis technique of multicast transmission with an additional auxiliary receiver, which serves as a genie to the original receiver. The…

The design and analysis of communication systems typically rely on the development of mathematical models that describe the underlying communication channel, which dictates the relationship between the transmitted and the received signals.…