Related papers: Ergodic MIMO Mutual Information: Twenty Years Afte…
We develop the use of mutual information (MI), a well-established metric in information theory, to interpret the inner workings of deep learning models. To accurately estimate MI from a finite number of samples, we present GMM-MI…
Factorizing low-rank matrices has many applications in machine learning and statistics. For probabilistic models in the Bayes optimal setting, a general expression for the mutual information has been proposed using heuristic statistical…
The mutual information of two random variables i and j with joint probabilities t_ij is commonly used in learning Bayesian nets as well as in many other fields. The chances t_ij are usually estimated by the empirical sampling frequency…
Estimation of mutual information between (multidimensional) real-valued variables is used in analysis of complex systems, biological systems, and recently also quantum systems. This estimation is a hard problem, and universally good…
Mutual information (MI) is a fundamental quantity in information theory and machine learning. However, direct estimation of MI is intractable, even if the true joint probability density for the variables of interest is known, as it involves…
We present ITE (information theoretical estimators) a free and open source, multi-platform, Matlab/Octave toolbox that is capable of estimating many different variants of entropy, mutual information, divergence, association measures, cross…
The mutual information is a measure of classical and quantum correlations of great interest in quantum information. It is also relevant in quantum many-body physics, by virtue of satisfying an area law for thermal states and bounding all…
The use of Mutual Information (MI) as a measure to evaluate the efficiency of cryptosystems has an extensive history. However, estimating MI between unknown random variables in a high-dimensional space is challenging. Recent advances in…
The mutual information is bounded from above by a decreasing affine function of the square of the distance between the input distribution and the set of all capacity-achieving input distributions $\Pi_{\mathcal{A}}$, on small enough…
In this letter, we investigate the mutual information rate (MIR) achieved by an independent identically distributed (IID) Gaussian input on the intensity-driven signal transduction channel. Specifically, the asymptotic expression of the…
In this paper, we derive Gallager's random coding error exponent for multiple-input multiple-output (MIMO) channels, assuming no channel-state information (CSI) at the transmitter and perfect CSI at the receiver. This measure gives insight…
We provide matching upper and lower bounds on the mutual information in noisy reconstruction of parity check codes and thereby prove a long-standing conjecture by Montanari [IEEE Transactions on Information Theory 2005]. Besides extending a…
We investigate the capacity scaling of MIMO systems with the system dimensions. To that end, we quantify how the mutual information varies when the number of antennas (at either the receiver or transmitter side) is altered. For a system…
In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to…
We consider the estimation of a signal from the knowledge of its noisy linear random Gaussian projections, a problem relevant in compressed sensing, sparse superposition codes or code division multiple access just to cite few. There has…
Mutual information is a general statistical dependency measure which has found applications in representation learning, causality, domain generalization and computational biology. However, mutual information estimators are typically…
One of the main notions of information theory is the notion of mutual information in two messages (two random variables in Shannon information theory or two binary strings in algorithmic information theory). The mutual information in $x$…
The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission…
We obtain ergodic theorems for multiple iterated sums and integrals of the form $\Sigma^{(\nu)}(t)=\sum_{0\leq k_1<...<k_\nu\leq t}\xi(k_1)\otimes\cdots\otimes\xi(k_\nu)$, $t\in[0,T]$ and $\Sigma^{(\nu)}(t)=\int_{0\leq s_1\leq...\leq…
This paper investigates the achievable sum rate of multiple-input multiple-output (MIMO) wireless systems employing linear minimum mean-squared error (MMSE) receivers. We present a new analytic framework which unveils an interesting…