Related papers: An entropy functional bounded from above by one
We study an entropy measure for quantum systems that generalizes the von Neumann entropy as well as its classical counterpart, the Gibbs or Shannon entropy. The entropy measure is based on hypothesis testing and has an elegant formulation…
The weak law of large numbers implies that, under mild assumptions on the source, the Renyi entropy per produced symbol converges (in probability) towards the Shannon entropy rate. This paper quantifies the speed of this convergence for…
A novel, non-trivial, probabilistic upper bound on the entropy of an unknown one-dimensional distribution, given the support of the distribution and a sample from that distribution, is presented. No knowledge beyond the support of the…
Shannon entropy for discrete distributions is a fundamental and widely used concept, but its continuous analogue, known as differential entropy, lacks essential properties such as positivity and compatibility with the discrete case. In this…
We derive an asymptotic lower bound on the Shannon entropy $H$ of sums of $N$ arbitrary iid discrete random variables. The derived bound $H \geq \frac{r(X)}{2}\log(N) + {\it cst}$ is given in terms of the incommensurability rank $r(X)$ of…
The concept of entropy, firstly introduced in information theory, rapidly became popular in many applied sciences via Shannon's formula to measure the degree of heterogeneity among observations. A rather recent research field aims at…
Even if a probability distribution is properly normalizable, its associated Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze conditions under which this phenomenon can occur. Roughly speaking, this happens when…
The Shannon entropy, and related quantities such as mutual information, can be used to quantify uncertainty and relevance. However, in practice, it can be difficult to compute these quantities for arbitrary probability distributions,…
It is not obvious how to extend Shannon's original information entropy to higher dimensions, and many different approaches have been tried. We replace the English text symbol sequence originally used to illustrate the theory by a discrete,…
Shannon's entropy is one of the building blocks of information theory and an essential aspect of Machine Learning methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable…
The quality of image encryption is commonly measured by the Shannon entropy over the ciphertext image. However, this measurement does not consider to the randomness of local image blocks and is inappropriate for scrambling based image…
We prove a Bernstein-type bound for the difference between the average of negative log-likelihoods of independent discrete random variables and the Shannon entropy, both defined on a countably infinite alphabet. The result holds for the…
A method for estimating the Shannon differential entropy of multidimensional random variables using independent samples is described. The method is based on decomposing the distribution into a product of the marginal distributions and the…
In this paper, I expand Shannon's definition of entropy into a new form of entropy that allows integration of information from different random events. Shannon's notion of entropy is a special case of my more general definition of entropy.…
We introduce a new measure of interdependence among the components of a random vector along the main diagonal of the vector copula, i.e. along the line $u_{1}=\ldots=u_{J}$, for $\left(u_{1},\ldots,u_{J}\right)\in\left[0,1\right]^{J}$. Our…
Entropy is a measure of heterogeneity widely used in applied sciences, often when data are collected over space. Recently, a number of approaches has been proposed to include spatial information in entropy. The aim of entropy is to…
Claude Shannon coined entropy to quantify the uncertainty of a random distribution for communication coding theory. We observe that the uncertainty nature of entropy also limits its direct usage in mathematical modeling. Therefore we…
We consider infinitely dimensional classes of functions and instead of the relative error setting, which was used in previous papers on the integral norm discretization, we consider the absolute error setting. We demonstrate how known…
We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the…
Complementarity relations between various characterizations of a probability distribution are at the core of information theory. In particular, lower and upper bounds for the entropic function are of great importance. In applied topics, we…