Related papers: Entropy and the Law of Small Numbers
An information-theoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Let $P_{S_n}$ be the distribution of a sum $S_n=\Sumn Y_i$ of…
The first part of this work considers the entropy of the sum of (possibly dependent and non-identically distributed) Bernoulli random variables. Upper bounds on the error that follows from an approximation of this entropy by the entropy of…
This paper considers the entropy of the sum of (possibly dependent and non-identically distributed) Bernoulli random variables. Upper bounds on the error that follows from an approximation of this entropy by the entropy of a Poisson random…
New families of Fisher information and entropy power inequalities for sums of independent random variables are presented. These inequalities relate the information in the sum of $n$ independent random variables to the information contained…
We describe five types of results concerning information and concentration of discrete random variables, and relationships between them, motivated by their counterparts in the continuous case. The results we consider are information…
This paper derives new bounds on the difference of the entropies of two discrete random variables in terms of the local and total variation distances between their probability mass functions. The derivation of the bounds relies on maximal…
The classical Poisson theorem says that if $\xi_1,\xi_2,...$ are i.i.d. 0--1 Bernoulli random variables taking on 1 with probability $p_n\equiv \la/n$ then the sum $S_n=\sum_{i=1}^n\xi_i$ is asymptotically in $n$ Poisson distributed with…
We establish a lower bound on the entropy of weighted sums of (possibly dependent) random variables $(X_1, X_2, \dots, X_n)$ possessing a symmetric joint distribution. Our lower bound is in terms of the joint entropy of $(X_1, X_2, \dots,…
The well-known "Janson's inequality" gives Poisson-like upper bounds for the lower tail probability \Pr(X \le (1-\eps)\E X) when X is the sum of dependent indicator random variables of a special form. We show that, for large deviations,…
Renyi's "thinning" operation on a discrete random variable is a natural discrete analog of the scaling operation for continuous random variables. The properties of thinning are investigated in an information-theoretic context, especially in…
Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as…
An "entropy increasing to the maximum" result analogous to the entropic central limit theorem (Barron 1986; Artstein et al. 2004) is obtained in the discrete setting. This involves the thinning operation and a Poisson limit. Monotonic…
In this paper we use a Malliavin-Stein type method to investigate Poisson and normal approximations for the measurable functions of infinitely many independent random variables. We combine Stein's method with the difference operators in…
We use the Stein-Chen method to prove new explicit inequalities for the total variation, Wasserstein and local distances between the distribution of a random diagonal sum of a Bernoulli matrix and a Poisson distribution. Approximation…
Let $X_1,X_2,...,X_n$ be a sequence of independent or locally dependent random variables taking values in $\mathbb{Z}_+$. In this paper, we derive sharp bounds, via a new probabilistic method, for the total variation distance between the…
Shearer's inequality bounds the sum of joint entropies of random variables in terms of the total joint entropy. We give another lower bound for the same sum in terms of the individual entropies when the variables are functions of…
We consider the central limit theorem for stable laws in the case of the standardized sum of independent and identically distributed random variables with regular probability density function. By showing decay of different entropy…
Let $X_1,\ldots,X_n$ be a sequence of independent random points in $\mathbb{R}^d$ with common Lebesgue density $f$. Under some conditions on $f$, we obtain a Poisson limit theorem, as $n \to \infty$, for the number of large probability…
Various lower bounds are established for the entropy of sums, products and their combinations. First, we derive a prime-field analogue of a version of the entropy power inequality established by Tao over torsion-free groups. Next, we prove…
One of the difficulties in calculating the capacity of certain Poisson channels is that H(lambda), the entropy of the Poisson distribution with mean lambda, is not available in a simple form. In this work we derive upper and lower bounds…