Related papers: An Effective Bernstein-type Bound on Shannon Entro…
We introduce the concept of an extremely negatively dependent (END) sequence of random variables with a given common marginal distribution. The END structure, as a new benchmark for negative dependence, is comparable to comonotonicity and…
Rule based classifiers that use the presence and absence of key sub-strings to make classification decisions have a natural mechanism for quantifying the uncertainty of their precision. For a binary classifier, the key insight is to treat…
The uneven distribution of digits in numerical data, known as Benford's law, was discovered in 1881. Since then, this law has been shown to be correct in copious numerical data relating to economics, physics and even prime numbers. Although…
In quantum Shannon theory, various kinds of quantum entropies are used to characterize the capacities of noisy physical systems. Among them, min-entropy and its smooth version attract wide interest especially in the field of quantum…
The Shannon Noiseless coding theorem (the data-compression principle) asserts that for an information source with an alphabet $\mathcal X=\{0,\ldots ,\ell -1\}$ and an asymptotic equipartition property, one can reduce the number of stored…
We show that the probability that a multilinear polynomial $f$ of independent random variables exceeds its mean by $\lambda$ is at most $e^{-\lambda^2 / (R^q Var(f))}$ for sufficiently small $\lambda$, where $R$ is an absolute constant.…
We present a detailed derivation of some estimators of Shannon entropy for discrete distributions. They hold for finite samples of N points distributed into M "boxes", with N and M -> oo, but N/M < oo. In the high sampling regime (<< 1…
Entropy and differential entropy are important quantities in information theory. A tractable extension to singular random variables-which are neither discrete nor continuous-has not been available so far. Here, we present such an extension…
Let $(d_n)$ be a sequence of positive numbers and let $(X_n)$ be a sequence of positive independent random variables. We provide an upper bound for the deviation between the distribution of the mantissaes of $(X_n^{d_n})$ and the Benford's…
We prove a tight uniform continuity bound for the conditional Shannon entropy of discrete finitely supported random variables in terms of total variation distance.
Shannon entropy, a cornerstone of information theory, statistical physics and inference methods, is uniquely identified by the Shannon-Khinchin or Shore-Johnson axioms. Generalizations of Shannon entropy, motivated by the study of…
A Poisson Binomial distribution over $n$ variables is the distribution of the sum of $n$ independent Bernoullis. We provide a sample near-optimal algorithm for testing whether a distribution $P$ supported on $\{0,...,n\}$ to which we have…
We introduce unbiased estimators for the Shannon entropy and the class number, in the situation that we are able to take sequences of independent samples of arbitrary length.
Classical (or ``global'') Bernstein theory establishes sharp control on entire functions of exponential type that are bounded and real-valued on the real axis. We localize some of this theory to rectangular regions $\{ x+iy: x \in I, 0 \leq…
We explore the relation between entanglement entropy of quantum many body systems and the distribution of corresponding, properly selected, observables. Such a relation is necessary to actually measure the entanglement entropy. We show that…
Let $X$ be an absolutely continuous random variable from the integrated Pearson family and assume that $X$ has finite moments of any order. Using some properties of the associated orthonormal polynomial system, we provide a class of…
In the present paper we prove a family of tight upper and lower bounds for the Shannon entropy and von Neumann entropy based on the p-norms. This allows us to have an entropy estimate, a criterion for the finiteness of it and a bound on the…
For a polynomial $P_n$ of degree $n$, Bernstein's inequality states that $\|P_n'\| \le n \|P_n\|$ for all $L^p$ norms on the unit circle, $0<p\le\infty,$ with equality for $P_n(z)= c z^n.$ We study this inequality for random polynomials,…
In this paper we obtain a Bernstein type inequality for a class of weakly dependent and bounded random variables. The proofs lead to a moderate deviations principle for sums of bounded random variables with exponential decay of the strong…
We show that for log-concave real random variables with fixed variance the Shannon differential entropy is minimized for an exponential random variable. We apply this result to derive upper bounds on capacities of additive noise channels…