Related papers: Lower bound on Wyner's Common Information
One of the basic tenets in information theory, the data processing inequality states that output divergence does not exceed the input divergence for any channel. For channels without input constraints, various estimates on the amount of…
The conditional mean is a fundamental and important quantity whose applications include the theories of estimation and rate-distortion. It is also notoriously difficult to work with. This paper establishes novel bounds on the differential…
The generalization error of a learning algorithm refers to the discrepancy between the loss of a learning algorithm on training data and that on unseen testing data. Various information-theoretic bounds on the generalization error have been…
The model considered is that of ``signal plus white noise.'' Known connections between the noncausal filtering error and mutual information are combined with new ones involving the causal estimation error, in a general abstract setup. The…
Information-theoretic generalization bounds analyze stochastic optimization by relating expected generalization error to the mutual information between learned parameters and training data. Virtual perturbation analyses of SGD add auxiliary…
We consider the prediction of weak effects in a multiple-output regression setup, when covariates are expected to explain a small amount, less than $\approx 1%$, of the variance of the target variables. To facilitate the prediction of the…
New upper bounds on the sum capacity of the two-user Gaussian interference channel are derived. Using these bounds, it is shown that treating interference as noise achieves the sum capacity if the interference levels are below certain…
An information-theoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Let $P_{S_n}$ be the distribution of a sum $S_n=\Sumn Y_i$ of…
We provide a lower bound on the probability that a binomial random variable is exceeding its mean. Our proof employs estimates on the mean absolute deviation and the tail conditional expectation of binomial random variables.
There have been several upper bounds on the quantum capacity of the single-mode Gaussian channels with thermal noise, such as thermal attenuator and amplifier. We consider a class of attenuator and amplifier with more general noises,…
We consider multiple access communication on a binary input additive white Gaussian noise channel using randomly spread code division. For a general class of symmetric distributions for spreading coefficients, in the limit of a large number…
This paper studies the two-user Causal Cognitive Interference Channel (CCIC), where two transmitters aim to communicate independent messages to two different receivers via a common channel. One source, referred to as the cognitive, is…
Distributed consensus in the Wasserstein metric space of probability measures on the real line is introduced in this work. Convergence of each agent's measure to a common measure is proven under a weak network connectivity condition. The…
In this paper, we generalize the fundamental relation between the derivative of the mutual information and the minimum mean squared error (MMSE) to multiuser setups. We prove that the derivative of the mutual information with respect to the…
Tight bounds on the minimum mean square error for the additive Gaussian noise channel are derived, when the input distribution is constrained to be epsilon-close to a Gaussian reference distribution in terms of the Kullback--Leibler…
Given two channels that convey information about the same random variable, we introduce two measures of the unique information of one channel with respect to the other. The two quantities are based on the notion of generalized weighted Le…
Recovery of the causal structure of dynamic networks from noisy measurements has long been a problem of interest across many areas of science and engineering. Many algorithms have been proposed, but there is little work that compares the…
Multilayer (or deep) networks are powerful probabilistic models based on multiple stages of a linear transform followed by a non-linear (possibly random) function. In general, the linear transforms are defined by matrices and the non-linear…
The mutual information is bounded from above by a decreasing affine function of the square of the distance between the input distribution and the set of all capacity-achieving input distributions $\Pi_{\mathcal{A}}$, on small enough…
A continuous-time Wiener phase noise channel with an integrate-and-dump multi-sample receiver is studied. A lower bound to the capacity with an average input power constraint is derived, and a high signal-to-noise ratio (SNR) analysis is…