Minimax Optimal Estimation of KL Divergence for Continuous Distributions
Information Theory
2020-02-27 v1 math.IT
Machine Learning
Abstract
Estimating Kullback-Leibler divergence from identical and independently distributed samples is an important problem in various domains. One simple and effective estimator is based on the k nearest neighbor distances between these samples. In this paper, we analyze the convergence rates of the bias and variance of this estimator. Furthermore, we derive a lower bound of the minimax mean square error and show that kNN method is asymptotically rate optimal.
Keywords
Cite
@article{arxiv.2002.11599,
title = {Minimax Optimal Estimation of KL Divergence for Continuous Distributions},
author = {Puning Zhao and Lifeng Lai},
journal= {arXiv preprint arXiv:2002.11599},
year = {2020}
}