Related papers: Monotonically Decreasing Sequence of Divergences
Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been…
We examine the convergence properties of sequences of nonnegative real numbers that satisfy a particular class of recursive inequalities, from the perspective of proof theory and computability theory. We first establish a number of results…
Convergent sequences of real numbers play a fundamental role in many different problems in system theory, e.g., in Lyapunov stability analysis, as well as in optimization theory and computational game theory. In this survey, we provide an…
The Kullback-Leibler (KL) divergence is a foundational measure for comparing probability distributions. Yet in multivariate settings, its single value often obscures the underlying reasons for divergence, conflating mismatches in individual…
Score-matching generative models have proven successful at sampling from complex high-dimensional data distributions. In many applications, this distribution is believed to concentrate on a much lower $d$-dimensional manifold embedded into…
The distribution of the spacing, or the difference between consecutive order statistics, is known only for uniform and exponential random variates. We add here logistic and Gumbel variates, and present an estimator for distributions with a…
We formulate a new information-theoretic principle--the shifted composition rule--which bounds the divergence (e.g., Kullback-Leibler or R\'enyi) between the laws of two stochastic processes via the introduction of auxiliary shifts. In this…
We study concentration inequalities for the Kullback--Leibler (KL) divergence between the empirical distribution and the true distribution. Applying a recursion technique, we improve over the method of types bound uniformly in all regimes…
Many studies have been conducted on statistical convergence, and it remains an area of active research. Since its introduction, statistical convergence has found applications many fields. Nevertheless, there is a shortage of research…
There are three classical divergence measures in the literature on information theory and statistics, namely, Jeffryes-Kullback-Leiber's J-divergence, Sibson-Burbea-Rao's Jensen-Shannon divegernce and Taneja's arithemtic-geometric mean…
Some concepts, such as non-compactness measure and condensing operators, defined on metric spaces are extended to uniform spaces. Such extensions allow us to locate, in the context of uniform spaces, some classical results existing in…
Estimating the Kullback-Leibler (KL) divergence between two distributions given samples from them is well-studied in machine learning and information theory. Motivated by considerations of multi-group fairness, we seek KL divergence…
This article develops an analytical framework for studying information divergences and likelihood ratios associated with Poisson processes and point patterns on general measurable spaces. The main results include explicit analytical…
Divergences, also known as contrast functions, are distance-like quantities defined on manifolds of non-negative or probability measures. Using the duality in optimal transport, we introduce and study the one-parameter family of $L^{(\pm…
The Pinsker inequality lower bounds the Kullback--Leibler divergence $D_{\textrm{KL}}$ in terms of total variation and provides a canonical way to convert $D_{\textrm{KL}}$ control into $\lVert \cdot \rVert_1$-control. Motivated by…
Diffusive molecular dynamics is a novel model for materials with atomistic resolution that can reach diffusive time scales. The main ideas of diffusive molecular dynamics are to first minimize an approximate variational Gaussian free energy…
There are three classical divergence measures known in the literature on information theory and statistics. These are namely, Jeffryes-Kullback-Leiber \cite{jef} \cite{kul} \textit{J-divergence}. Sibson-Burbea-Rao \cite{sib} \cite{bur1,…
We consider continuous Dirac operators defined on $\mathbf{R}^d$, $d\in\{1,2,3\}$, together with various discrete versions of them. Both forward-backward and symmetric finite differences are used as approximations to partial derivatives. We…
A derivation operator and a divergence operator are defined on the algebra of bounded operators on the symmetric Fock space over the complexification of a real Hilbert space $\eufrak{h}$ and it is shown that they satisfy similar properties…
The ability to compute the exact divergence between two high-dimensional distributions is useful in many applications but doing so naively is intractable. Computing the alpha-beta divergence -- a family of divergences that includes the…