Related papers: Entropy and a generalisation of `Poincare's Observ…
We give a new characterization of relative entropy, also known as the Kullback-Leibler divergence. We use a number of interesting categories related to probability theory. In particular, we consider a category FinStat where an object is a…
We show that the moment generating function of the Kullback-Leibler divergence (relative entropy) between the empirical distribution of $n$ independent samples from a distribution $P$ over a finite alphabet of size $k$ (i.e. a multinomial…
In this note, we show that the relative entropy of an empirical distribution of $n$ samples drawn from a set of size $k$ with respect to the true underlying distribution is exponentially concentrated around its expectation, with central…
Relative entropy is a fundamental class of distances between probability distributions, with widespread applications in probability theory, statistics, and machine learning. In this work, we study relative entropy from a categorical…
We define a general notion of entropy in elementary, algebraic terms. Based on that, weak forms of a scalar product and a distance measure are derived. We give basic properties of these quantities, generalize the Cauchy-Schwarz inequality,…
We prove a fractional version of Poincar\'e inequalities in the context of $\R^n$ endowed with a fairly general measure. Namely we prove a control of an $L^2$ norm by a non local quantity, which plays the role of the gradient in the…
We introduce the notion of relative volume entropy for two spacetimes with preferred compact spacelike foliations. This is accomplished by applying the notion of Kullback-Leibler divergence to the volume elements induced on spacelike…
In this paper, we investigate the partition inequality, joint convexity, and Pinsker's inequality, for a divergence that generalizes the Tsallis Relative Entropy and Kullback-Leibler divergence. The generalized divergence is defined in…
A generalized Kullback-Leibler relative entropy is introduced starting with the symmetric Jackson derivative of the generalized overlap between two probability distributions. The generalization retains much of the structure possessed by the…
Our aim is to provide a short and self contained synthesis which generalise and unify various related and unrelated works involving what we call Phi-Sobolev functional inequalities. Such inequalities related to Phi-entropies can be seen in…
Formalising the confrontation of opinions (models) to observations (data) is the task of Inferential Statistics. Information Theory provides us with a basic functional, the relative entropy (or Kullback-Leibler divergence), an asymmetrical…
We revisit entropic formulations of the uncertainty principle for an arbitrary pair of positive operator-valued measures (POVM) $A$ and $B$, acting on finite dimensional Hilbert space. Salicr\'u generalized $(h,\phi)$-entropies, including…
The Poincare constant R(Y) of a random variable Y relates the L2 norm of a function g and its derivative g'. Since R(Y) - Var(Y) is positive, with equality if and only if Y is normal, it can be seen as a distance from the normal…
A finite form of de Finetti's representation theorem is established using elementary information-theoretic tools: The distribution of the first $k$ random variables in an exchangeable binary vector of length $n\geq k$ is close to a mixture…
This paper provides a quantitative version of de Finetti law of large numbers. Given an infinite sequence $\{X_n\}_{n \geq 1}$ of exchangeable Bernoulli variables, it is well-known that $\frac{1}{n} \sum_{i = 1}^n X_i…
In this paper, we provide the proof of $L^2$ consistency for the $k$th nearest neighbour distance estimator of the Shannon entropy for an arbitrary fixed $k\geq 1.$ We construct the non-parametric test of goodness-of-fit for a class of…
We observe some higher order Poincare-type inequalities on a closed manifold, which is inspired by Hurwitz's proof of the Wirtinger's inequality using Fourier theory. We then give some geometric implication of these inequalities by applying…
The statistical measure of spatial inhomogeneity for n points placed in chi cells each of size kxk is generalized to incorporate finite size objects like black pixels for binary patterns of size LxL. As a function of length scale k, the…
Under Poincar\'e-type conditions, upper bounds are explored for the Kolmogorov distance between the distributions of weighted sums of dependent summands and the normal law. Based on improved concentration inequalities on high-dimensional…
New upper bounds on the relative entropy are derived as a function of the total variation distance. One bound refines an inequality by Verd\'{u} for general probability measures. A second bound improves the tightness of an inequality by…