Related papers: On the Kullback-Leibler divergence between pairwis…
We derive a closed form solution for the Kullback-Leibler divergence between two Weibull distributions. These notes are meant as reference material and intended to provide a guided tour towards a result that is often mentioned but seldom…
We obtain formulas for Petz-R\'enyi and Umegaki relative entropy from the idea of distribution of a positive selfadjoint operator. Classical results on R\'enyi and Kullback-Leibler divergences are applied to obtain new results and new…
The entanglement entropy of a subsystem of a quantum system is expressed, in the replica approach, through analytic continuation with respect to n of the trace of the n-th power of the reduced density matrix. This trace can be thought of as…
To characterize the Kullback-Leibler divergence and Fisher information in general parametrized hidden Markov models, in this paper, we first show that the log likelihood and its derivatives can be represented as an additive functional of a…
We study the problem of estimating a distribution over a finite alphabet from an i.i.d. sample, with accuracy measured in relative entropy (Kullback-Leibler divergence). While optimal bounds on the expected risk are known, high-probability…
The irreversibility of a stationary time series can be quantified using the Kullback-Leibler divergence (KLD) between the probability to observe the series and the probability to observe the time-reversed series. Moreover, this KLD is a…
This paper introduces two new robust methods for estimation of parameters in a given parametric family. The first method is that of `minimum weighted L2', effectively minimising an estimate of the integrated (and possibly weighted) squared…
In this paper, some new upper bounds for Kullback-Leibler divergence(KL-divergence) based on $L^1, L^2$ and $L^\infty$ norms of density functions are discussed. Our findings unveil that the convergence in KL-divergence sense sandwiches…
Mixture distributions arise in many parametric and non-parametric settings -- for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in most cases, this…
The study of two-dimensional Coulomb gases lies at the interface of statistical physics and non-Hermitian random matrix theory. In this paper we give a large deviation principle (LDP) for the empirical fields obtained, under the canonical…
In information theory -- as well as in the adjacent fields of statistics, machine learning, artificial intelligence, signal processing and pattern recognition -- many flexibilizations of the omnipresent Kullback-Leibler information distance…
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the…
Multivariate datasets are common in various real-world applications. Recently, copulas have received significant attention for modeling dependencies among random variables. A copula-based information measure is required to quantify the…
We study the Kullback--Leibler (KL) divergence approximation theory of Gaussian mixture models (GMMs) by isolating an abstract mechanism behind several necessary-and-sufficient statements. The necessity direction is universal: if a density…
Most popular dimension reduction (DR) methods like t-SNE and UMAP are based on minimizing a cost between input and latent pairwise similarities. Though widely used, these approaches lack clear probabilistic foundations to enable a full…
We apply two variations of the principle of Minimum Cross Entropy (the Kullback information measure) to fit parameterized probability density models to observed data densities. For an array beamforming problem with P incident narrowband…
Similarity-sensitive entropy measures the uncertainty of a probability law relative to a similarity kernel that encodes the distinguishability between states. We develop a measure-theoretic treatment covering both finite similarity matrices…
Distance covariance is a quantity to measure the dependence of two random vectors. We show that the original concept introduced and developed by Sz\'{e}kely, Rizzo and Bakirov can be embedded into a more general framework based on symmetric…
Model or variable selection is usually achieved through ranking models according to the increasing order of preference. One of methods is applying Kullback-Leibler distance or relative entropy as a selection criterion. Yet that will raise…
We use the $f-divergence$ also called relative entropy as a measure of diversity between probability densities and review its basic properties. In the sequence we define a few objects which capture relevant information from the sample of a…