Related papers: $H(X)$ vs. $H(f(X))$

It is well known that the entropy $H(X)$ of a discrete random variable $X$ is always greater than or equal to the entropy $H(f(X))$ of a function $f$ of $X$, with equality if and only if $f$ is one-to-one. In this paper, we give tight…

Various lower bounds are established for the entropy of sums, products and their combinations. First, we derive a prime-field analogue of a version of the entropy power inequality established by Tao over torsion-free groups. Next, we prove…

The entropy power inequality (EPI) provides lower bounds on the differential entropy of the sum of two independent real-valued random variables in terms of the individual entropies. Versions of the EPI for discrete random variables have…

Let $\mathsf{N}_{\rm d}\left[X\right]=\frac{1}{2\pi {\rm e}}{\rm e}^{2\mathsf{H}\left[X\right]}$ denote the entropy power of the discrete random variable $X$ where $\mathsf{H}\left[X\right]$ denotes the discrete entropy of $X$. In this…

A novel, non-trivial, probabilistic upper bound on the entropy of an unknown one-dimensional distribution, given the support of the distribution and a sample from that distribution, is presented. No knowledge beyond the support of the…

We establish a lower bound on the entropy of weighted sums of (possibly dependent) random variables $(X_1, X_2, \dots, X_n)$ possessing a symmetric joint distribution. Our lower bound is in terms of the joint entropy of $(X_1, X_2, \dots,…

A bound for functional $\Delta(F)=\sup_{x\in\mathbb R}|F(x)-\Phi(x)|$ is obtained, which is uniform for all distribution functions $F$ of random variables with zero mean-value and unity variance. Moreover, a two-point distribution is found,…

Let $X$ denote a nonnegative random variable with $\mathsf{E} X<\infty$. Upper and lower bounds on $\mathsf{E} X-\exp\mathsf{E}\ln X$ are obtained, which are exact, in terms of $V_X$ and $E_X$ for the upper bound and in terms of $V_X$ and…

Any coded subshift X defined by a set C of code words contains a subshift, which we call L, consisting of limits of single code words. We show that when C satisfies a unique decomposition property, the topological entropy h(X) of X is…

Building on the recent work of Johnson (2007) and Yu (2008), we prove that entropy is a concave function with respect to the thinning operation T_a. That is, if X and Y are independent random variables on Z_+ with ultra-log-concave…

Let $X$ be a random variable with distribution function $F,$ and $X_{1},X_{2},...,X_{n}$ are independent copies of $X.$ Consider the order statistics $X_{i:n},$ $i=1,2,...,n$ and denote $F_{i:n}(x)=P\{X_{i:n}\leq x\}.$ Using majorization…

We revisit the problem of the estimation of the differential entropy $H(f)$ of a random vector $X$ in $R^d$ with density $f$, assuming that $H(f)$ exists and is finite. In this note, we study the consistency of the popular nearest neighbor…

We study the relative entropy between the empirical estimate of a discrete distribution and the true underlying distribution. If the minimum value of the probability mass function exceeds an $\alpha > 0$ (i.e. when the true underlying…

The weighted entropy $H^{\rm w}_\phi (X)=H^{\rm w}_\phi (f)$ of a random variable $X$ with values $x$ and a probability-mass/density function $f$ is defined as the mean value ${\mathbb E} I^{\rm w}_\phi(X)$ of the weighted information…

We prove that there is no algorithm to decide whether a given integer vector is in the closure of the entropic cone $\overline{\Gamma_{n}^{*}}$. Equivalently, there is no decision procedure to determine whether a given integer-valued…

Entropic uncertainty relations $H(A)+H(B)\geqslant \gamma$ give a nonzero lower bound $\gamma$ to the sum of the Shannon entropies $H$ of the outcome probabilities of incompatible observables $A$ and $B$. They are better than the…

We give the proof of a tight lower bound on the probability that a binomial random variable exceeds its expected value. The inequality plays an important role in a variety of contexts, including the analysis of relative deviation bounds in…

We prove a lower bound on the relative entropy between two finite-dimensional states in terms of their entropy difference and the dimension of the underlying space. The inequality is tight in the sense that equality can be attained for any…

The exact lower bound on the probability of the occurrence of exactly one of $n$ random events each of probability $p$ is obtained.

We derive an asymptotic lower bound on the Shannon entropy $H$ of sums of $N$ arbitrary iid discrete random variables. The derived bound $H \geq \frac{r(X)}{2}\log(N) + {\it cst}$ is given in terms of the incommensurability rank $r(X)$ of…