Related papers: Shannon's entropy revisited
It is well known that to estimate the Shannon entropy for symbolic sequences accurately requires a large number of samples. When some aspects of the data are known it is plausible to attempt to use this to more efficiently compute entropy.…
In this paper, I expand Shannon's definition of entropy into a new form of entropy that allows integration of information from different random events. Shannon's notion of entropy is a special case of my more general definition of entropy.…
Reliable data-driven estimation of Shannon entropy from small data sets, where the number of examples is potentially smaller than the number of possible outcomes, is a critical matter in several applications. In this paper, we introduce a…
The concept of entropy, firstly introduced in information theory, rapidly became popular in many applied sciences via Shannon's formula to measure the degree of heterogeneity among observations. A rather recent research field aims at…
We explore the relation between entanglement entropy of quantum many body systems and the distribution of corresponding, properly selected, observables. Such a relation is necessary to actually measure the entanglement entropy. We show that…
Choice overload occurs when individuals feel overwhelmed by an excessive number of options. Experimental evidence suggests that a larger selection can complicate the decision-making process. Consequently, choice satisfaction may diminish…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
Many of the traditional results in information theory, such as the channel coding theorem or the source coding theorem, are restricted to scenarios where the underlying resources are independent and identically distributed (i.i.d.) over a…
It is not obvious how to extend Shannon's original information entropy to higher dimensions, and many different approaches have been tried. We replace the English text symbol sequence originally used to illustrate the theory by a discrete,…
We consider the entropy of sums of independent discrete random variables, in analogy with Shannon's Entropy Power Inequality, where equality holds for normals. In our case, infinite divisibility suggests that equality should hold for…
This paper introduces an objective metric for evaluating a parsing scheme. It is based on Shannon's original work with letter sequences, which can be extended to part-of-speech tag sequences. It is shown that this regular language is an…
Shannon's information entropy measures of the uncertainty of an event's outcome. If learning about a system reflects a decrease in uncertainty, then a plausible intuition is that learning should be accompanied by a decrease in the entropy…
After Shannon, entropy becomes a fundamental quantity to describe not only uncertainity or chaos of a system but also information carried by the system. Shannon's important discovery is to give a mathematical expression of the mutual…
Information plays an important role in our understanding of the physical world. We hence propose an entropic measure of information for any physical theory that admits systems, states and measurements. In the quantum and classical world,…
We revisit the well-studied problem of estimating the Shannon entropy of a probability distribution, now given access to a probability-revealing conditional sampling oracle. In this model, the oracle takes as input the representation of a…
Shannon's metric of "Entropy" of information is a foundational concept of information theory. This article is a primer for novices that presents an intuitive way of understanding, remembering, and/or reconstructing Shannon's Entropy metric…
Statements of Shannon's Noiseless Coding Theorem by various authors, including the original, are reviewed and clarified. Traditional statements of the theorem are often unclear as to when it applies. A new notation is introduced and the…
Shannon entropy was defined for probability distributions and then its using was expanded to measure the uncertainty of knowledge for systems with complete information. In this article, it is proposed to extend the using of Shannon entropy…
Entropic measures of complexity are able to quantify the information encoded in complex network structures. Several entropic measures have been proposed in this respect. Here we study the relation between the Shannon entropy and the Von…
In this study, the cumulative effect of the empirical probability distribution of a random variable is identified as a factor that amplifies the occurrence of extreme events in datasets. To quantify this observation, a corresponding…