Related papers: Shannon Entropy Estimation in $\infty$-Alphabets f…
This paper considers the estimation of Shannon entropy for discrete distributions with countably infinite support. While minimax rates for finite-support distributions are established, infinite-support distributions present distinct…
Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language. However, entropy must typically be estimated from observed data because researchers do not have access to the underlying…
Shannon's entropy is one of the building blocks of information theory and an essential aspect of Machine Learning methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable…
We consider the problem of finite sample corrections for entropy estimation. New estimates of the Shannon entropy are proposed and their systematic error (the bias) is computed analytically. We find that our results cover correction…
We present a detailed derivation of some estimators of Shannon entropy for discrete distributions. They hold for finite samples of N points distributed into M "boxes", with N and M -> oo, but N/M < oo. In the high sampling regime (<< 1…
We present estimators for entropy and other functions of a discrete probability distribution when the data is a finite sample drawn from that probability distribution. In particular, for the case when the probability distribution is a joint…
Shannon entropy is widely used to quantify the uncertainty of discrete random variables. But when normalized to the unit interval, as is often done in practice, it no longer conveys the alphabet sizes of the random variables being studied.…
Shannon's entropy and other entropy-based concepts are derived from the new, more general concept of relative divergence of one "grading' function on a linearly ordered set from another such function. The definition of relative divergence…
We introduce unbiased estimators for the Shannon entropy and the class number, in the situation that we are able to take sequences of independent samples of arbitrary length.
The behavior of the Kozachenko - Leonenko estimates for the (differential) Shannon entropy is studied when the number of i.i.d. vector-valued observations tends to infinity. The asymptotic unbiasedness and L^2-consistency of the estimates…
We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our…
Convergence properties of Shannon Entropy are studied. In the differential setting, it is shown that weak convergence of probability measures, or convergence in distribution, is not enough for convergence of the associated differential…
In the present paper, we consider the plug-in estimator of Shannon's entropy defined on a finite alphabet which is assumed to dynamically vary as the sample size increases. The asymptotic behaviors for the plug-in estimator, such as,…
We have presented a new axiomatic derivation of Shannon Entropy for a discrete probability distribution on the basis of the postulates of additivity and concavity of the entropy function.We have then modified shannon entropy to take account…
Shannon entropy was defined for probability distributions and then its using was expanded to measure the uncertainty of knowledge for systems with complete information. In this article, it is proposed to extend the using of Shannon entropy…
Shannon Entropy is the preeminent tool for measuring the level of uncertainty (and conversely, information content) in a random variable. In the field of communications, entropy can be used to express the information content of given…
Even if a probability distribution is properly normalizable, its associated Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze conditions under which this phenomenon can occur. Roughly speaking, this happens when…
The statistical analysis of data stemming from dynamical systems, including, but not limited to, time series, routinely relies on the estimation of information theoretical quantities, most notably Shannon entropy. To this purpose, possibly…
For studies in reliability, biometry, and survival analysis, the length-biased distribution is often well-suited for certain natural sampling plans. In this paper, we study the strong uniform consistency of two nonparametric estimators for…
The Shannon entropy, and related quantities such as mutual information, can be used to quantify uncertainty and relevance. However, in practice, it can be difficult to compute these quantities for arbitrary probability distributions,…