Related papers: Computing Entropies With Nested Sampling
The concept of entropy, firstly introduced in information theory, rapidly became popular in many applied sciences via Shannon's formula to measure the degree of heterogeneity among observations. A rather recent research field aims at…
Understanding the way in which random entities interact is of key interest in numerous scientific fields. This can range from a full characterization of the joint distribution to single scalar summary statistics. In this work we identify a…
Even if a probability distribution is properly normalizable, its associated Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze conditions under which this phenomenon can occur. Roughly speaking, this happens when…
We describe a method to computationally estimate the probability density function of a univariate random variable by applying the maximum entropy principle with some local conditions given by Gaussian functions. The estimation errors and…
The mutual information (MI) between two random variables is an important correlation measure in data analysis. The Shannon entropy of a joint probability distribution is the variable part under fixed marginals. We aim to minimize and…
The organization of high-dimensional probability spaces is a fundamental problem at the intersection of statistical physics and information theory. Here, we analyze the distributions populating level surfaces of the probability simplex…
We deploy Shannon's information entropy to the distribution of branching fractions in a particle decay. This serves to quantify how important a given new reported decay channel is, from the point of view of the information that it adds to…
The Shannon entropy is a fundamental measure for quantifying diversity and model complexity in fields such as information theory, ecology, and genetics. However, many existing studies assume that the number of species is known, an…
After Shannon, entropy becomes a fundamental quantity to describe not only uncertainity or chaos of a system but also information carried by the system. Shannon's important discovery is to give a mathematical expression of the mutual…
The Shannon entropy, one of the cornerstones of information theory, is widely used in physics, particularly in statistical mechanics. Yet its characterization and connection to physics remain vague, leaving ample room for misconceptions and…
Nested sampling (NS) is a stochastic method for computing the log-evidence of a Bayesian problem. It relies on stochastic estimates of prior volumes enclosed by likelihood contours, which limits the accuracy of the log-evidence calculation.…
Algorithmic entropy and Shannon entropy are two conceptually different information measures, as the former is based on size of programs and the later in probability distributions. However, it is known that, for any recursive probability…
Estimating the Shannon entropy of a discrete distribution from which we have only observed a small sample is challenging. Estimating other information-theoretic metrics, such as the Kullback-Leibler divergence between two sparsely sampled…
We present a new class of estimators of Shannon entropy for severely undersampled discrete distributions. It is based on a generalization of an estimator proposed by T. Schuermann, which itself is a generalization of an estimator proposed…
Many inference problems involve inferring the number $N$ of components in some region, along with their properties $\{\mathbf{x}_i\}_{i=1}^N$, from a dataset $\mathcal{D}$. A common statistical example is finite mixture modelling. In the…
Nested sampling (NS) computes parameter posterior distributions and makes Bayesian model comparison computationally feasible. Its strengths are the unsupervised navigation of complex, potentially multi-modal posteriors until a well-defined…
Probability theory is fundamental for modeling uncertainty, with traditional probabilities being real and non-negative. Complex probability extends this concept by allowing complex-valued probabilities, opening new avenues for analysis in…
The quality of image encryption is commonly measured by the Shannon entropy over the ciphertext image. However, this measurement does not consider to the randomness of local image blocks and is inappropriate for scrambling based image…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
Nested sampling (NS) is a popular algorithm for Bayesian computation. We investigate statistical errors in NS both analytically and numerically. We show two analytic results. First, we show that the leading terms in Skilling's expression…