Related papers: Information Spectrum Converse for Minimum Entropy …
Given a collection of probability distributions $p_{1},\ldots,p_{m}$, the minimum entropy coupling is the coupling $X_{1},\ldots,X_{m}$ ($X_{i}\sim p_{i}$) with the smallest entropy $H(X_{1},\ldots,X_{m})$. While this problem is known to be…
Given two discrete random variables $X$ and $Y,$ with probability distributions ${\bf p}=(p_1, \ldots , p_n)$ and ${\bf q}=(q_1, \ldots , q_m)$, respectively, denote by ${\cal C}({\bf p}, {\bf q})$ the set of all couplings of ${\bf p}$ and…
We consider the problem of coding for computing with maximal distortion, where the sender communicates with a receiver, which has its own private data and wants to compute a function of their combined data with some fidelity constraint…
A coupling of two distributions $P_{X}$ and $P_{Y}$ is a joint distribution $P_{XY}$ with marginal distributions equal to $P_{X}$ and $P_{Y}$. Given marginals $P_{X}$ and $P_{Y}$ and a real-valued function $f$ of the joint distribution…
In relative entropy coding, a sender aims to design a stochastic code such that, on input $X \sim P_X$, the receiver can generate a sample $Y \sim P_{Y \mid X}$. It is a standard result that (1) this requires at least $I(X; Y)$ bits, (2)…
Given a set of discrete probability distributions, the minimum entropy coupling is the minimum entropy joint distribution that has the input distributions as its marginals. This has immediate relevance to tasks such as entropic causal…
In this paper, some general properties of Shannon information measures are investigated over sets of probability distributions with restricted marginals. Certain optimization problems associated with these functionals are shown to be…
Dependence among marginally constrained observations can break a finite-sample barrier. To formalize this phenomenon, we introduce the \emph{minimum list entropy coupling} $H(P\|Q_1,\dots,Q_m)$, the minimum conditional entropy…
Entropy and differential entropy are important quantities in information theory. A tractable extension to singular random variables-which are neither discrete nor continuous-has not been available so far. Here, we present such an extension…
Minimum divergence problems under integral constraints appear throughout statistics and probability, including sequential inference, bandit theory, and distributionally robust optimization. In many such settings, dual representations are…
Given two discrete random variables $X$ and $Y$, with probability distributions ${\bf p} =(p_1, \ldots , p_n)$ and ${\bf q}=(q_1, \ldots , q_m)$, respectively, denote by ${\cal C}({\bf p}, {\bf q})$ the set of all couplings of ${\bf p}$ and…
An information-theoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Let $P_{S_n}$ be the distribution of a sum $S_n=\Sumn Y_i$ of…
Exponential models of distributions are widely used in machine learning for classiffication and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we…
The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed…
We derive information-theoretic converses (i.e., lower bounds) for the minimum time required by any algorithm for distributed function computation over a network of point-to-point channels with finite capacity, where each node of the…
We study the problem of discovering the simplest latent variable that can make two observed discrete variables conditionally independent. The minimum entropy required for such a latent is known as common entropy in information theory. We…
Conditional mutual information is important in the selection and interpretation of graphical models. Its empirical version is well known as a generalised likelihood ratio test and that it may be represented as a difference in entropy. We…
It is well known that the entropy $H(X)$ of a discrete random variable $X$ is always greater than or equal to the entropy $H(f(X))$ of a function $f$ of $X$, with equality if and only if $f$ is one-to-one. In this paper, we give tight…
Information functionals allow to quantify the degree of randomness of a given probability distribution, either absolutely (through min/max entropy principles) or relative to a prescribed reference one. Our primary aim is to analyze the…
The paper introduces the recent results related to an entropy functional on trajectories of a controlled diffusion process, and the information path functional (IPF), analyzing their connections to the Kolmogorov's entropy, complexity and…