Related papers: Computing the Kullback-Leibler Divergence between …
The Kullback-Leibler divergence or relative entropy is an information-theoretic measure between statistical models that play an important role in measuring a distance between random variables. In the study of complex systems, random fields…
We present a derivation of the Kullback Leibler (KL)-Divergence (also known as Relative Entropy) for the von Mises Fisher (VMF) Distribution in $d$-dimensions.
We consider the fundamental problem of estimating a discrete distribution on a domain of size $K$ with high probability in Kullback-Leibler divergence. We provide upper and lower bounds on the minimax estimation rate, which show that the…
By calculating the Kullback-Leibler divergence between two probability measures belonging to different exponential families, we end up with a formula that generalizes the ordinary Fenchel-Young divergence. Inspired by this formula, we…
There exist two different versions of the Kullback-Leibler divergence (K-Ld) in Tsallis statistics, namely the usual generalized K-Ld and the generalized Bregman K-Ld. Problems have been encountered in trying to reconcile them. A condition…
Estimating the Kullback-Leibler (KL) divergence between two distributions given samples from them is well-studied in machine learning and information theory. Motivated by considerations of multi-group fairness, we seek KL divergence…
We study concentration inequalities for the Kullback--Leibler (KL) divergence between the empirical distribution and the true distribution. Applying a recursion technique, we improve over the method of types bound uniformly in all regimes…
The problem of filtering information from large correlation matrices is of great importance in many applications. We have recently proposed the use of the Kullback-Leibler distance to measure the performance of filtering algorithms in…
The $\alpha$-divergences include the well-known Kullback-Leibler divergence, Hellinger distance and $\chi^2$-divergence. In this paper, we derive differential and integral relations between the $\alpha$-divergences that are generalizations…
Weibull distribution has received a wide range of applications in engineering and science. The utility and usefulness of an estimator is highly subject to the field of practitioner's study. In practice users looking for their desired…
The Kullback-Leibler (KL) divergence is not a proper distance metric and does not satisfy the triangle inequality, posing theoretical challenges in certain practical applications. Existing work has demonstrated that KL divergence between…
We provide optimal lower and upper bounds for the augmented Kullback-Leibler divergence in terms of the augmented total variation distance between two probability measures defined on two Euclidean spaces having different dimensions. We call…
Two approaches are suggested to the definition of asymmetric generalized Weibull distribution. These approaches are based on the representation of the two-sided Weibull distributions as variance-mean normal mixtures or more general…
The probability density quantile (pdQ) carries essential information regarding shape and tail behavior of a location-scale family. Convergence of repeated applications of the pdQ mapping to the uniform distribution is investigated and new…
In this paper, we consider the problem of the estimation of a Weibull tail-coefficient. In particular, we propose a regression model, from which we derive a bias-reduced estimator. This estimator is based on a least-squares approach. The…
The hybrid censoring is a mixture of Type I and Type II censoring schemes. This paper presents the statistical inferences of the Inverse Weibull distribution when the data are Type-I hybrid censored. First we consider the maximum likelihood…
Weibull distribution is widely used in modelling health data. However, its lack of sufficient tail flexibility often results in poor fit in extreme events. We proposed another three-parameter extension of the Weibull distribution with…
Here I present the analytic form of two common distance metrics, the symmetrised Kullback-Leibler Divergence and the Kolmogorov-Smirnov statistic, as well as an extension of the Kolmogorov-Smirnov statistic for comparing theoretical gamma…
The goal of this short note is to discuss the relation between Kullback--Leibler divergence and total variation distance, starting with the celebrated Pinsker's inequality relating the two, before switching to a simple, yet (arguably) more…
We show that the Kullback-Leibler distance is a good measure of the statistical uncertainty of correlation matrices estimated by using a finite set of data. For correlation matrices of multivariate Gaussian variables we analytically…