Related papers: Generalized information entropies depending only o…
A common statistical situation concerns inferring an unknown distribution Q(x) from a known distribution P(y), where X (dimension n), and Y (dimension m) have a known functional relationship. Most commonly, n<m, and the task is relatively…
We examine the {combinatorial} or {probabilistic} definition ("Boltzmann's principle") of the entropy or cross-entropy function $H \propto \ln \mathbb{W}$ or $D \propto - \ln \mathbb{P}$, where $\mathbb{W}$ is the statistical weight and…
We review with a tutorial scope the information theory foundations of quantum statistical physics. Only a small proportion of the variables that characterize a system at the microscopic scale can be controlled, for both practical and…
Probability distributions having power-law tails are observed in a broad range of social, economic, and biological systems. We describe here a potentially useful common framework. We derive distribution functions $\{p_k\}$ for situations in…
We demonstrate that Shannon's information entropy and the thermodynamic entropy of Boltzmann and Gibbs are quantitatively equivalent for real condensed-matter systems. By interpreting atomic configurations as information sources, we compute…
The limit of small entropy production is reached in relaxing systems long after preparation, and in stationary driven systems in the limit of small driving power. Surprisingly, for extended systems this limit is not in general the…
Tsallis' non-extensive entropy is extended to incorporate the dependence on affinities between the microstates of a system. At the core of our construction of the extended entropy ($\mathcal{H}$) is the concept of the effective number of…
It is argued that the factorization of compound probability over subsystems is a consequence of the existence of thermodynamic equilibrium in the composite system having Tsallis entropy. So it should be respected by all exact calculations…
The combinatorial basis of entropy, given by Boltzmann, can be written $H = N^{-1} \ln \mathbb{W}$, where $H$ is the dimensionless entropy, $N$ is the number of entities and $\mathbb{W}$ is number of ways in which a given realization of a…
Regardless of studies and debates over a century, the statistical origin of the second law of thermodynamics still remains illusive. One essential obstacle is the lack of a proper theoretical formalism for non-equilibrium entropy. Here I…
The method of Maximum (relative) Entropy (ME) is used to translate the information contained in the known form of the likelihood into a prior distribution for Bayesian inference. The argument is guided by intuition gained from the…
Large-scale astrophysical systems are non-extensive due to their long-range force of gravity. Here we show an approach toward the statistical mechanics of such self-gravitating systems (SGS). This is a generalization of the standard…
For a dynamical system far from equilibrium, one has to deal with empirical probabilities defined through time-averages, and the main problem is then how to formulate an appropriate statistical thermodynamics. The common answer is that the…
Information theoretic measures (entropies, entropy rates, mutual information) are nowadays commonly used in statistical signal processing for real-world data analysis. The present work proposes the use of Auto Mutual Information (Mutual…
We propose a new method for the calculation of the statistical properties, as e.g. the entropy, of unknown generators of symbolic sequences. The probability distribution $p(k)$ of the elements $k$ of a population can be approximated by the…
We consider a bipartite quantum conductor and discuss the joint probability distribution of particle number in a subsystem and the self-information associated with the reduced density matrix of the subsystem. By extending the multi-contour…
Even if a probability distribution is properly normalizable, its associated Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze conditions under which this phenomenon can occur. Roughly speaking, this happens when…
Ergodicity, this is to say, dynamics whose time averages coincide with ensemble averages, naturally leads to Boltzmann-Gibbs (BG) statistical mechanics, hence to standard thermodynamics. This formalism has been at the basis of an enormous…
The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space,…
Mixture distributions are a workhorse model for multimodal data in information theory, signal processing, and machine learning. Yet even when each component density is simple, the differential entropy of the mixture is notoriously hard to…