An Effective Bernstein-type Bound on Shannon Entropy over Countably Infinite Alphabets
Information Theory
2021-06-24 v1 math.IT
Probability
Abstract
We prove a Bernstein-type bound for the difference between the average of negative log-likelihoods of independent discrete random variables and the Shannon entropy, both defined on a countably infinite alphabet. The result holds for the class of discrete random variables with tails lighter than or on the same order of a discrete power-law distribution. Most commonly-used discrete distributions such as the Poisson distribution, the negative binomial distribution, and the power-law distribution itself belong to this class. The bound is effective in the sense that we provide a method to compute the constants in it.
Keywords
Cite
@article{arxiv.2106.12149,
title = {An Effective Bernstein-type Bound on Shannon Entropy over Countably Infinite Alphabets},
author = {Yunpeng Zhao},
journal= {arXiv preprint arXiv:2106.12149},
year = {2021}
}