Related papers: Combinatorial Information Theory: I. Philosophical…
We examine the {combinatorial} or {probabilistic} definition ("Boltzmann's principle") of the entropy or cross-entropy function $H \propto \ln \mathbb{W}$ or $D \propto - \ln \mathbb{P}$, where $\mathbb{W}$ is the statistical weight and…
The combinatorial basis of entropy, given by Boltzmann, can be written $H = N^{-1} \ln \mathbb{W}$, where $H$ is the dimensionless entropy, $N$ is the number of entities and $\mathbb{W}$ is number of ways in which a given realization of a…
We give an overview of the role of information theory in statistics, and particularly in biostatistics. We recall the basic quantities in information theory; entropy, cross-entropy, conditional entropy, mutual information and…
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the…
A new combinatorial-probabilistic diagnostic entropy has been introduced. It describes the pair-wise sum of probabilities of system conditions that have to be distinguished during the diagnosing process. The proposed measure describes the…
Information theory is a mathematical theory of learning with deep connections with topics as diverse as artificial intelligence, statistical physics, and biological evolution. Many primers on information theory paint a broad picture with…
The combinatorial basis of entropy by Boltzmann can be written $H= {N}^{-1} \ln \mathbb{W}$, where $H$ is the dimensionless entropy of a system, per unit entity, $N$ is the number of entities and $\mathbb{W}$ is the number of ways in which…
A unified combinatorial definition of the information content and entropy of different types of patterns, compatible with the traditional concepts of information and entropy, going beyond the limitations of Shannon information interpretable…
Probability theory is fundamental for modeling uncertainty, with traditional probabilities being real and non-negative. Complex probability extends this concept by allowing complex-valued probabilities, opening new avenues for analysis in…
Quantum information theory, particularly its entropic formulations, has made remarkable strides in characterizing quantum systems and tasks. However, a critical dimension remains underexplored: computational efficiency. While classical…
Some general considerations on the notion of entropy in physics are presented. An attempt is made to clarify the question of the differentiation between physical entropy (the Clausius-Boltzmann one) and quantities called entropies…
Complex systems that are characterized by strong correlations and fat-tailed distribution functions have been argued to be incompatible within the framework of Boltzmann-Gibbs entropy. As an alternative, so-called generalized entropies were…
Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a…
Many complex systems are characterized by non-Boltzmann distribution functions of their statistical variables. If one wants to -- justified or not -- hold on to the maximum entropy principle for complex statistical systems (non-Boltzmann)…
We propose a new interpretation of measures of information and disorder by connecting these concepts to group theory in a new way. Entropy and group theory are connected here by their common relation to sets of permutations. A combinatorial…
"Bounds on information combining" are entropic inequalities that determine how the information (entropy) of a set of random variables can change when these are combined in certain prescribed ways. Such bounds play an important role in…
We investigate the concept of entropy in probabilistic theories more general than quantum mechanics, with particular reference to the notion of information causality recently proposed by Pawlowski et. al. (arXiv:0905.2992). We consider two…
The logical basis for information theory is the newly developed logic of partitions that is dual to the usual Boolean logic of subsets. The key concept is a "distinction" of a partition, an ordered pair of elements in distinct blocks of the…
According to E.T. Jaynes and E.P. Wigner, entropy is an anthropomorphic concept in the sense that in a physical system correspond many thermodynamic systems. The physical system can be examined from many points of view each time examining…
The maximum entropy principle (MEP) is a method for obtaining the most likely distribution functions of observables from statistical systems, by maximizing entropy under constraints. The MEP has found hundreds of applications in ergodic and…