Related papers: Statistical estimation of the Kullback-Leibler div…
Relative entropy is a fundamental class of distances between probability distributions, with widespread applications in probability theory, statistics, and machine learning. In this work, we study relative entropy from a categorical…
Data collection is a critical step in statistical inference and data science, and the goal of statistical experimental design (ED) is to find the data collection setup that can provide most information for the inference. In this work we…
It has been argued persuasively that, in order to evaluate climate models, the probability distributions of model output need to be compared to the corresponding empirical distributions of observed data. Distance measures between…
This paper deals with a method for the approximation of a spectral density function among the solutions of a generalized moment problem a` la Byrnes/Georgiou/Lindquist. The approximation is pursued with respect to the Kullback-Leibler…
We develop hypothesis testing for active information -the averaged quantity in the Kullback-Liebler divergence. To our knowledge, this is the first paper to derive exact probabilities of type-I errors for hypothesis testing in the area.
A method for selecting a graphical model for $p$-vector-valued stationary Gaussian time series was recently proposed by Matsuda and uses the Kullback-Leibler divergence measure to define a test statistic. This statistic was used in a…
We consider the fundamental problem of estimating a discrete distribution on a domain of size $K$ with high probability in Kullback-Leibler divergence. We provide upper and lower bounds on the minimax estimation rate, which show that the…
We review recent results about the maximal values of the Kullback-Leibler information divergence from statistical models defined by neural networks, including naive Bayes models, restricted Boltzmann machines, deep belief networks, and…
We conduct a KL-divergence based procedure for testing elliptical distributions. The procedure simultaneously takes into account the two defining properties of an elliptically distributed random vector: independence between length and…
Divergences are quantities that measure discrepancy between two probability distributions and play an important role in various fields such as statistics and machine learning. Divergences are non-negative and are equal to zero if and only…
The Pinsker inequality lower bounds the Kullback--Leibler divergence $D_{\textrm{KL}}$ in terms of total variation and provides a canonical way to convert $D_{\textrm{KL}}$ control into $\lVert \cdot \rVert_1$-control. Motivated by…
In this paper, we provide the proof of $L^2$ consistency for the $k$th nearest neighbour distance estimator of the Shannon entropy for an arbitrary fixed $k\geq 1.$ We construct the non-parametric test of goodness-of-fit for a class of…
In this paper, we derive some upper and lower bounds and inequalities for the total variation distance (TVD) and the Kullback-Leibler divergence (KLD), also known as the relative entropy, between two probability measures $\mu$ and $\nu$…
In this paper we derive an integral (with respect to time) representation of the relative entropy (or Kullback-Leibler Divergence) between measures mu and P on the space of continuous functions from time 0 to T. The underlying measure P is…
The deepening penetration of renewable resources into power systems entails great difficulties that have not been surmounted satisfactorily. An issue that merits special attention is the short-term planning of power systems under net load…
We consider the parameter estimation problem of a probabilistic generative model prescribed using a natural exponential family of distributions. For this problem, the typical maximum likelihood estimator usually overfits under limited…
In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean Square Error (wMSE)…
Estimating the Kullback-Leibler (KL) divergence between two distributions given samples from them is well-studied in machine learning and information theory. Motivated by considerations of multi-group fairness, we seek KL divergence…
Maximum regularized likelihood estimators (MRLEs) are arguably the most established class of estimators in high-dimensional statistics. In this paper, we derive guarantees for MRLEs in Kullback-Leibler divergence, a general measure of…
The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space,…