Related papers: Variability as a better characterization of Shanno…
Entropy has emerged as a dynamic, interdisciplinary, and widely accepted quantitative measure of uncertainty across different disciplines. A unified understanding of entropy measures, supported by a detailed review of their theoretical…
Despite the wide usage of information as a concept in science, we have yet to develop a clear & concise scientific definition. This paper is aimed at laying the foundations for a new theory concerning the mechanics of information alongside…
The problems of causality, modeling, and control for chaotic, high-dimensional dynamical systems are formulated in the language of information theory. The central quantity of interest is the Shannon entropy, which measures the amount of…
Shannon's entropy is one of the building blocks of information theory and an essential aspect of Machine Learning methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable…
Statistical physics aims to describe properties of macroscale systems in terms of distributions of their microscale agents. Its central tool is the maximization of entropy, a variational principle. We review the history of this principle,…
It is pointed out that the case for Shannon entropy and von Neumann entropy, as measures of uncertainty in quantum mechanics, is not as bleak as suggested in quant-ph/0006087. The main argument of the latter is based on one particular…
Shannon entropy is widely used to quantify the uncertainty of discrete random variables. But when normalized to the unit interval, as is often done in practice, it no longer conveys the alphabet sizes of the random variables being studied.…
An information theory description of finite systems explicitly evolving in time is presented for classical as well as quantum mechanics. We impose a variational principle on the Shannon entropy at a given time while the constraints are set…
Uncertainty relations are central to quantum physics. While they were originally formulated in terms of variances, they have later been successfully expressed with entropies following the advent of Shannon information theory. Here, we…
The quantification of the complexity of networks is, today, a fundamental problem in the physics of complex systems. A possible roadmap to solve the problem is via extending key concepts of information theory to networks. In this paper we…
The participation coefficient is a widely used metric of the diversity of a node's connections with respect to a modular partition of a network. An information-theoretic formulation of this concept of connection diversity, referred to here…
Accurately determining dependency structure is critical to discovering a system's causal organization. We recently showed that the transfer entropy fails in a key aspect of this---measuring information flow---due to its conflation of dyadic…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
Thermodynamic entropy, as defined by Clausius, characterizes macroscopic observations of a system based on phenomenological quantities such as temperature and heat. In contrast, information-theoretic entropy, introduced by Shannon, is a…
Probability theory is fundamental for modeling uncertainty, with traditional probabilities being real and non-negative. Complex probability extends this concept by allowing complex-valued probabilities, opening new avenues for analysis in…
The notion of entropy penetrates much of science. A key feature of the all-important notion of Boltzmann-Gibbs-Shannon entropy is its extensivity (additivity over independent subsystems). However, there is a need for other quantities. In…
The Shannon entropy is a fundamental measure for quantifying diversity and model complexity in fields such as information theory, ecology, and genetics. However, many existing studies assume that the number of species is known, an…
After Shannon, entropy becomes a fundamental quantity to describe not only uncertainity or chaos of a system but also information carried by the system. Shannon's important discovery is to give a mathematical expression of the mutual…
The formalism of statistical mechanics can be generalized by starting from more general measures of information than the Shannon entropy and maximizing those subject to suitable constraints. We discuss some of the most important examples of…
The chain rule for the Shannon and von Neumann entropy, which relates the total entropy of a system to the entropies of its parts, is of central importance to information theory. Here we consider the chain rule for the more general smooth…