Related papers: Computing the Kullback-Leibler Divergence between …
In a variety of applications it is important to extract information from a probability measure $\mu$ on an infinite dimensional space. Examples include the Bayesian approach to inverse problems and possibly conditioned) continuous time…
The $\alpha$-divergences include the well-known Kullback-Leibler divergence, Hellinger distance and $\chi^2$-divergence. In this paper, we derive differential and integral relations between the $\alpha$-divergences that are generalizations…
Estimating the Kullback-Leibler (KL) divergence between two distributions given samples from them is well-studied in machine learning and information theory. Motivated by considerations of multi-group fairness, we seek KL divergence…
We consider the problem of sampling from a probability distribution $\pi$ which admits a density w.r.t. a dominating measure. It is well known that this can be written as an optimisation problem over the space of probability distributions…
The goal of this short note is to discuss the relation between Kullback--Leibler divergence and total variation distance, starting with the celebrated Pinsker's inequality relating the two, before switching to a simple, yet (arguably) more…
Generative models have achieved remarkable success across a range of applications, yet their evaluation still lacks principled uncertainty quantification. In this paper, we develop a method for comparing how close different generative…
In a regression setup with deterministic design, we study the pure aggregation problem and introduce a natural extension from the Gaussian distribution to distributions in the exponential family. While this extension bears strong…
In this paper, we prove Raabe-type integral formulas for gamma function via left and right sided Riemann-Liouville fractional integrals. As corollaries, we give the left and right sided repeated integration formulas for the log-gamma and…
The partial differential equation of Gaussian diffusion is generalized by using the time-fractional derivative of distributed order between 0 and 1, in both the Riemann-Liouville (R-L) and the Caputo (C) sense. For a general distribution of…
A rigorous connection between large deviations theory and Gamma-convergence is established. Applications include representations formulas for rate functions, a contraction principle for measurable maps, a large deviations principle for…
The theory of normal variance mixture distributions is used to provide elementary derivations of closed-form expressions for the definite integrals $\int_0^\infty x^{-2\nu}\cos(bx)\gamma(\nu,\alpha x^2)\,\mathrm{d}x$ (for $\nu>1/2$, $b>0$…
In this paper, we derive a useful lower bound for the Kullback-Leibler divergence (KL-divergence) based on the Hammersley-Chapman-Robbins bound (HCRB). The HCRB states that the variance of an estimator is bounded from below by the…
Estimating the Kullback-Leibler (KL) divergence between random variables is a fundamental problem in statistical analysis. For continuous random variables, traditional information-theoretic estimators scale poorly with dimension and/or…
We establish isomorphism ranges for the comparison maps between algebraic and topological K-groups, extending classical Quillen-Lichtenbaum conjecture to separated complex schemes of finite type after refinement. Additionally, we…
$f$-divergences, which quantify discrepancy between probability distributions, are ubiquitous in information theory, machine learning, and statistics. While there are numerous methods for estimating $f$-divergences from data, a limit…
We derive a Dickman approximation for the small jumps of a large class of multivariate L\'evy processes. We then apply this approximation to develop a simulation method for the class of general multivariate gamma distributions (GMGD). A…
In this paper, certain generalized fractional derivative formulae are introduced involving the k-Mittag-Leffler function. Then their image formulae (using Beta transform, Laplace transform and Whittaker transform) are also established. The…
We provide optimal lower and upper bounds for the augmented Kullback-Leibler divergence in terms of the augmented total variation distance between two probability measures defined on two Euclidean spaces having different dimensions. We call…
The non-parametric version of Amari's dually affine Information Geometry provides a practical calculus to perform computations of interest in statistical machine learning. The method uses the notion of a statistical bundle, a mathematical…
The Kullback-Leibler (KL) divergence is a foundational measure for comparing probability distributions. Yet in multivariate settings, its single value often obscures the underlying reasons for divergence, conflating mismatches in individual…