Related papers: Entropy Measures vs. Algorithmic Information
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
According to E.T. Jaynes and E.P. Wigner, entropy is an anthropomorphic concept in the sense that in a physical system correspond many thermodynamic systems. The physical system can be examined from many points of view each time examining…
I consider the effect of a finite sample size on the entropy of a sample of independent events. I propose formula for entropy which satisfies Shannon's axioms, and which reduces to Shannon's entropy when sample size is infinite. I discuss…
The paper examines relationships between the Shannon entropy and the $\ell_{\alpha}$-norm for $n$-ary probability vectors, $n \ge 2$. More precisely, we investigate the tight bounds of the $\ell_{\alpha}$-norm with a fixed Shannon entropy,…
The formalism of statistical mechanics can be generalized by starting from more general measures of information than the Shannon entropy and maximizing those subject to suitable constraints. We discuss some of the most important examples of…
The paper describes an approach to measuring convergence of an algorithm to its result in terms of an entropy-like function of partitions of its inputs of a given length. The goal is to look at the algorithmic data processing from the…
We find the value of constants related to constraints in characterization of some known statistical distributions and then we proceed to use the idea behind maximum entropy principle to derive generalized version of this distributions using…
Whereas Shannon entropy is related to the growth rate of multinomial coefficients, we show that the quadratic entropy (Tsallis 2-entropy) is connected to their $q$-deformation; when $q$ is a prime power, these $q$-multinomial coefficients…
Permutation entropy quantifies the diversity of possible orderings of the values a random or deterministic system can take, as Shannon entropy quantifies the diversity of values. We show that the metric and permutation entropy…
In information theory, one major goal is to find useful functions that summarize the amount of information contained in the interaction of several random variables. Specifically, one can ask how the classical Shannon entropy, mutual…
It is shown that the standard expression for the information entropy, originally due to Shannon, is only valid for a particular set of states. For the general case of statistical mechanics, one needs to include an additional term in the…
It is not obvious how to extend Shannon's original information entropy to higher dimensions, and many different approaches have been tried. We replace the English text symbol sequence originally used to illustrate the theory by a discrete,…
A comparative study of one-dimensional quantum structures which allow analytic expressions for the position and momentum R\'{e}nyi $R(\alpha)$ and Tsallis $T(\alpha)$ entropies, focuses on extracting the most characteristic physical…
Shannon's entropy is one of the building blocks of information theory and an essential aspect of Machine Learning methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable…
Entropy is a fundamental property of both classical and quantum systems, spanning myriad theoretical and practical applications in physics and computer science. We study the problem of obtaining estimates to within a multiplicative factor…
Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated…
The ultimate purpose of the statistical analysis of ordinal patterns is to characterize the distribution of the features they induce. In particular, knowing the joint distribution of the pair Entropy-Statistical Complexity for a large class…
Entropy is a measure of self-information which is used to quantify losses. Entropy was developed in thermodynamics, but is also used to compare probabilities based on their deviating information content. Corresponding model uncertainty is…
In this paper, we consider the information content of maximum ranked set sampling procedure with unequal samples (MRSSU) in terms of Tsallis entropy which is a nonadditive generalization of Shannon entropy. We obtain several results of…
This paper reveals a conceptually new connection from information theory to approximation theory via quantum algorithms for entropy estimation. Specifically, we provide an information-theoretic lower bound $\Omega(\sqrt{n})$ on the…