Related papers: Jaynes & Shannon's Constrained Ignorance and Surpr…
This paper introduces an objective metric for evaluating a parsing scheme. It is based on Shannon's original work with letter sequences, which can be extended to part-of-speech tag sequences. It is shown that this regular language is an…
It is pointed out that the case for Shannon entropy and von Neumann entropy, as measures of uncertainty in quantum mechanics, is not as bleak as suggested in quant-ph/0006087. The main argument of the latter is based on one particular…
Observational entropy captures both the intrinsic uncertainty of a thermodynamic state and the lack of knowledge due to coarse-graining. We demonstrate two interpretations of observational entropy, one as the statistical deficiency…
E.T. Jaynes, originator of the maximum entropy interpretation of statistical mechanics, emphasized that there is an inevitable trade-off between the conflicting requirements of robustness and accuracy for any inferencing algorithm. This is…
The paper suggests the concepts of an upper entropy and a lower entropy. We propose a new axiomatic definition, namely, upper entropy axioms, inspired by axioms of metric spaces, and also formulate lower entropy axioms. We also develop weak…
This paper considers the estimation of Shannon entropy for discrete distributions with countably infinite support. While minimax rates for finite-support distributions are established, infinite-support distributions present distinct…
Information and uncertainty are closely related and extensively studied concepts in a number of scientific disciplines such as communication theory, probability theory, and statistics. Increasing the information arguably reduces the…
The behavior of the Kozachenko - Leonenko estimates for the (differential) Shannon entropy is studied when the number of i.i.d. vector-valued observations tends to infinity. The asymptotic unbiasedness and L^2-consistency of the estimates…
In this paper, a simple case of Bayesian mechanics under the free energy principle is formulated in axiomatic terms. We argue that any dynamical system with constraints on its dynamics necessarily looks as though it is performing inference…
We consider two types of entropy, namely, Shannon and R\'{e}nyi entropies of the Poisson distribution, and establish their properties as the functions of intensity parameter. More precisely, we prove that both entropies increase with…
The general idea of information entropy provided by C.E. Shannon "hangs over everything we do" and can be applied to a great variety of problems once the connection between a distribution and the quantities of interest is found. The Shannon…
The ``Gibbs Paradox'' refers to several related questions concerning entropy in thermodynamics and statistical mechanics: whether it is an extensive quantity or not, how it changes when identical particles are mixed, and the proper way to…
We revisit the well-studied problem of estimating the Shannon entropy of a probability distribution, now given access to a probability-revealing conditional sampling oracle. In this model, the oracle takes as input the representation of a…
In [Gallego and Aolita, Physical Review X 5, 041008 (2015)], the authors proposed a definition for the relative entropy of steering and showed that the resulting quantity is a convex steering monotone. Here we advocate for a different…
The paper deals with the generalization of both Boltzmann entropy and distribution in the light of most-probable interpretation of statistical equilibrium. The statistical analysis of the generalized entropy and distribution leads to some…
R\'enyi entropy is a one-parameter generalization of Shannon entropy, which has been used in various fields of physics. Despite its wide applicability, the physical interpretations of the R\'enyi entropy are not widely known. In this paper,…
We discuss how the method of maximum entropy, MaxEnt, can be extended beyond its original scope, as a rule to assign a probability distribution, to a full-fledged method for inductive inference. The main concept is the (relative) entropy…
We consider a simplified version of a solvable model by Mandal and Jarzynski, which constructively demonstrates the interplay between work extraction and the increase of the Shannon entropy of an information reservoir which is in contact…
It is well-known that the partition function can consistently be factorized from the canonical equilibrium distribution obtained through the maximization of the Shannon entropy. We show that such a normalized and factorized equilibrium…
Maximum entropy principle identifies forces conjugated to observables and the thermodynamic relations between them, independent upon their underlying mechanistic details. For data about state distributions or transition statistics, the…