Related papers: From $\epsilon$-entropy to KL-entropy: Analysis of…
The necessary information to distinguish a local inhomogeneous mass density field from its spatial average on a compact domain of the universe can be measured by relative information entropy. The Kullback-Leibler (KL) formula arises very…
The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space,…
We examine the estimation of the Kullback-Leibler (KL) divergence and the use of the goodness-of-fit test for multivariate continuous distributions. Our starting point is the maximum entropy principle for Shannon entropy: among all…
We propose an entropy-based information measure, namely the Discounted Least Information Theory of Entropy (DLITE), which not only exhibits important characteristics expected as an information measure but also satisfies conditions of a…
Estimating the Shannon entropy of a discrete distribution from which we have only observed a small sample is challenging. Estimating other information-theoretic metrics, such as the Kullback-Leibler divergence between two sparsely sampled…
We extend previously proposed measures of complexity, emergence, and self-organization to continuous distributions using differential entropy. This allows us to calculate the complexity of phenomena for which distributions are known. We…
Formalising the confrontation of opinions (models) to observations (data) is the task of Inferential Statistics. Information Theory provides us with a basic functional, the relative entropy (or Kullback-Leibler divergence), an asymmetrical…
Estimating entropy and mutual information consistently is important for many machine learning applications. The Kozachenko-Leonenko (KL) estimator (Kozachenko & Leonenko, 1987) is a widely used nonparametric estimator for the entropy of…
As a part of the construction of an information theory based on general probabilistic theories, we propose and investigate the several distinguishability measures and "entropies" in general probabilistic theories. As their applications,…
We present an information-theoretic assessment of atomic and molecular densities in the ground state and under a range of physical scenarios--excitation, confinement, and ensemblization. Comparisons across densities obtained from…
We present novel bounds for estimating discrete probability distributions under the $\ell_\infty$ norm. These are nearly optimal in various precise senses, including a kind of instance-optimality. Our data-dependent convergence guarantees…
Entropy estimation is of practical importance in information theory and statistical science. Many existing entropy estimators suffer from fast growing estimation bias with respect to dimensionality, rendering them unsuitable for…
We study concentration inequalities for the Kullback--Leibler (KL) divergence between the empirical distribution and the true distribution. Applying a recursion technique, we improve over the method of types bound uniformly in all regimes…
We apply two variations of the principle of Minimum Cross Entropy (the Kullback information measure) to fit parameterized probability density models to observed data densities. For an array beamforming problem with P incident narrowband…
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the…
We consider the problem of estimating probability density functions based on sample data, using a finite mixture of densities from some component class. To this end, we introduce the $h$-lifted Kullback--Leibler (KL) divergence as a…
In this paper, some new upper bounds for Kullback-Leibler divergence(KL-divergence) based on $L^1, L^2$ and $L^\infty$ norms of density functions are discussed. Our findings unveil that the convergence in KL-divergence sense sandwiches…
In this paper we introduce a nonextensive quantum information theoretic measure which may be defined between any arbitrary number of density matrices, and we analyze its fundamental properties in the spectral graph-theoretic framework.…
Information theory provides principled ways to analyze different inference and learning problems such as hypothesis testing, clustering, dimensionality reduction, classification, among others. However, the use of information theoretic…
Considering the difficulty of interpreting generative model output, there is significant current research focused on determining meaningful evaluation metrics. Several recent approaches utilize "precision" and "recall," borrowed from the…