Related papers: Jaynes & Shannon's Constrained Ignorance and Surpr…
In this second article, we show a simple use of the Ignorance as defined in "Jaynes & Shannon's Constrained Ignorance and Surprise". By giving an example about the journey of a person, we believe to show some simple, obvious but…
Shannon entropy, a cornerstone of information theory, statistical physics and inference methods, is uniquely identified by the Shannon-Khinchin or Shore-Johnson axioms. Generalizations of Shannon entropy, motivated by the study of…
Shannon entropy was defined for probability distributions and then its using was expanded to measure the uncertainty of knowledge for systems with complete information. In this article, it is proposed to extend the using of Shannon entropy…
We have presented a new axiomatic derivation of Shannon Entropy for a discrete probability distribution on the basis of the postulates of additivity and concavity of the entropy function.We have then modified shannon entropy to take account…
Shannon entropy is widely used to quantify the uncertainty of discrete random variables. But when normalized to the unit interval, as is often done in practice, it no longer conveys the alphabet sizes of the random variables being studied.…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
In this paper, I expand Shannon's definition of entropy into a new form of entropy that allows integration of information from different random events. Shannon's notion of entropy is a special case of my more general definition of entropy.…
We show that the Jaynes principle is indeed a proper inference scheme when applied to compound systems and will correctly produce the entangled maximum entropy states compatible with appropriate data. This is accomplished by including the…
Shannon's entropy and other entropy-based concepts are derived from the new, more general concept of relative divergence of one "grading' function on a linearly ordered set from another such function. The definition of relative divergence…
The Shannon entropy, one of the cornerstones of information theory, is widely used in physics, particularly in statistical mechanics. Yet its characterization and connection to physics remain vague, leaving ample room for misconceptions and…
The Principle of Maximum Entropy is a rigorous technique for estimating an unknown distribution given partial information while simultaneously minimizing bias. However, an important requirement for applying the principle is that the…
Even if a probability distribution is properly normalizable, its associated Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze conditions under which this phenomenon can occur. Roughly speaking, this happens when…
The concept of entropy, firstly introduced in information theory, rapidly became popular in many applied sciences via Shannon's formula to measure the degree of heterogeneity among observations. A rather recent research field aims at…
The fundamentals of the Maximum Entropy principle as a rule for assigning and updating probabilities are revisited. The Shannon-Jaynes relative entropy is vindicated as the optimal criterion for use with an updating rule. A constructive…
Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as…
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible constrained to match empirical data, for instance, feature expectations. We seek to generalize…
According to E.T. Jaynes and E.P. Wigner, entropy is an anthropomorphic concept in the sense that in a physical system correspond many thermodynamic systems. The physical system can be examined from many points of view each time examining…
Within a framework of utmost generality, we show that the entropy maximization procedure with linear constraints uniquely leads to the Shannon-Boltzmann-Gibbs entropy. Therefore, the use of this procedure with linear constraints should not…
We demonstrate and discuss the process of gaining information and show an example in which some specific way of gaining information about an object results in the Tsallis form of entropy rather than in the Shannon one.
The problem of Shannon entropy estimation in countable infinite alphabets is addressed from the study and use of convergence results of the entropy functional, which is known to be discontinuous with respect to the total variation distance…