Related papers: On Finite-Time Mutual Information
Since the celebrated works of Russo and Zou (2016,2019) and Xu and Raginsky (2017), it has been well known that the generalization error of supervised learning algorithms can be bounded in terms of the mutual information between their input…
Shannon defined the mutual information between two variables. We illustrate why the true mutual information between a variable and the predictions made by a prediction algorithm is not a suitable measure of prediction quality, but the…
We consider three capacity definitions for general channels with channel side information at the receiver, where the channel is modeled as a sequence of finite dimensional conditional distributions not necessarily stationary, ergodic, or…
A standard assumption in the design of ultra-reliable low-latency communication systems is that the duration between message arrivals is larger than the number of channel uses before the decoding deadline. Nevertheless, this assumption…
Using the Ryu-Takayanagi conjectured formula for entanglement entropy in the context of gauge-gravity duality, we investigate properties of mutual information between two disjoint rectangular sub-systems in finite temperature relativistic…
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of…
We prove the Courtade-Kumar conjecture, which states that the mutual information between any Boolean function of an $n$-dimensional vector of independent and identically distributed inputs to a memoryless binary symmetric channel and the…
Exploration tasks are embedded in many robotics applications, such as search and rescue and space exploration. Information-based exploration algorithms aim to find the most informative trajectories by maximizing an information-theoretic…
The conditional mutual information I(X;Y|Z) measures the average information that X and Y contain about each other given Z. This is an important primitive in many learning problems including conditional independence testing, graphical model…
The design of error-correcting codes used in modern communications relies on information theory to quantify the capacity of a noisy channel to send information [1]. This capacity can be expressed using the mutual information between input…
We consider a multi-agent system where each agent has its own estimate of a given quantity and the goal is to reach consensus on the average. To this purpose, we propose a distributed consensus algorithm that guarantees convergence to the…
In the first chapter of Shannon's "A Mathematical Theory of Communication," it is shown that the maximum entropy rate of an input process of a constrained system is limited by the combinatorial capacity of the system. Shannon considers…
The information theoretic quantity known as mutual information finds wide use in classification and community detection analyses to compare two classifications of the same set of objects into groups. In the context of classification…
Correlations disguised in various forms underlie a host of important phenomena in classical and quantum systems, such as information and energy exchanges. The quantum mutual information and the norm of the correlation matrix are both…
MIMO processing techniques in fiber optical communications have been proposed as a promising approach to meet increasing demand for information throughput. In this context, the multiple channels correspond to the multiple modes and/or…
The capacity of a quantum channel for transmission of classical information depends in principle on whether product states or entangled states are used at the input, and whether product or entangled measurements are used at the output. We…
We model the transmission of a message on the complete graph with n vertices and limited resources. The vertices of the graph represent servers that may broadcast the message at random. Each server has a random emission capital that…
A central question in information theory is to determine the maximum success probability that can be achieved in sending a fixed number of messages over a noisy channel. This was first studied in the pioneering work of Shannon who…
Two novel views are presented on the trapdoor channel. First, by deriving the underlying iterated function system (IFS), it is shown that the trapdoor channel with input blocks of length $n$ can be regarded as the $n$th element of a…
This paper introduces time into information theory, gives a more accurate definition of information, and unifies the information in cognition and Shannon information theory. Specially, we consider time as a measure of information, giving a…