Related papers: Maxwell Construction: The Hidden Bridge between It…

We consider communication over binary-input memoryless output-symmetric channels using low-density parity-check codes and message-passing decoding. The asymptotic (in the length) performance of such a combination for a fixed number of…

Modular exponentiation is crucial to number theory and cryptography, yet remains largely unexplored from a mechanistic interpretability standpoint. We train a 4-layer encoder-decoder Transformer model to perform this operation and…

Recursive max-linear vectors provide models for causal dependence between large values of random variables that are supported on directed acyclic graphs, but the standard assumption that all nodes of such a graph are observed can be…

Algorithm extraction aims to synthesize executable programs directly from models trained on algorithmic tasks, enabling de novo algorithm discovery without relying on human-written code. However, applying this paradigm to Transformer is…

In this work, we present an alternative to conventional residual connections, which is inspired by maxout nets. This means that instead of the addition in residual connections, our approach only propagates the maximum value or, in the leaky…

In this paper an interpolation-based decoding algorithm to decode Gabidulin codes, transmitted through a finely restricted channel, is proposed. The algorithm is able to decode rank errors beyond half the minimum distance by one unit. Also…

The driving force behind deep networks is their ability to compactly represent rich classes of functions. The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one…

The generalization of Shannon's theory to include messages with given autocorrelations is presented. The analytical calculation of the channel capacity is based on the transfer matrix method of the effective 1D Hamiltonian. This bridge…

Multilayer (or deep) networks are powerful probabilistic models based on multiple stages of a linear transform followed by a non-linear (possibly random) function. In general, the linear transforms are defined by matrices and the non-linear…

Signal transduction in living cells is vital to maintain life itself, where information transfer in noisy environment plays a significant role. In a rather different context, the recent intensive researches of "Maxwell's demon" - a feedback…

The lack of uniqueness arising by oversampling of Fourier coefficients is shown to provide a way of transmitting hidden information. A basic encoding/decoding system, developed on the basis of such a possibility, is discussed. The system is…

In many systems consisting of interacting subsystems, the complex interactions between elements can be represented using multilayer networks. However percolation, key to understanding connectivity and robustness, is not trivially…

A central challenge in cellular signal processing is understanding how biochemical networks perform reliably despite molecular noise. Traditionally, mutual information has been widely used to quantify signaling fidelity, capturing how well…

This paper investigates the error probability of a stochastic decision and the way in which it differs from the error probability of an optimal decision, i.e., the maximum a posteriori decision. This paper calls attention to the fact that…

Optimization methods play a central role in signal processing, serving as the mathematical foundation for inference, estimation, and control. While classical iterative optimization algorithms provide interpretability and theoretical…

Dense prediction tasks have enjoyed a growing complexity of encoder architectures, decoders, however, have remained largely the same. They rely on individual blocks decoding intermediate feature maps sequentially. We introduce banks, shared…

We focus on belief propagation for the assignment problem, also known as the maximum weight bipartite matching problem. We provide a constructive proof that the well-known upper bound on the number of iterations (Bayati, Shah, Sharma 2008)…

We derive a tight lower bound on equivocation (conditional entropy), or equivalently a tight upper bound on mutual information between a signal variable and channel outputs. The bound is in terms of the joint distribution of the signals and…

Sampling from the posterior is a key technical problem in Bayesian statistics. Rigorous guarantees are difficult to obtain for Markov Chain Monte Carlo algorithms of common use. In this paper, we study an alternative class of algorithms…

This paper considers a binary channel with deletions and insertions, where each input bit is transformed in one of the following ways: it is deleted with probability d, or an extra bit is added after it with probability i, or it is…