Related papers: Generalized Mutual Information
Mutual information is widely used in artificial intelligence, in a descriptive way, to measure the stochastic dependence of discrete random variables. In order to address questions such as the reliability of the empirical value, one must…
This paper fills a gap in our understanding of the interaction between information and computation. It unifies other approaches to measuring information like Kolmogorov complexity and Shannon information. We define a theory about…
Motivation: Clustering is a frequently used concept in variety of bioinformatical applications. We present a new method for hierarchical clustering of data called mutual information clustering (MIC) algorithm. It uses mutual information…
Mutual information is widely used, in a descriptive way, to measure the stochastic dependence of categorical random variables. In order to address questions such as the reliability of the descriptive value, one must consider…
We address three outstanding problems in information theory. Problem one is the definition of a non-negative decomposition of the information conveyed by two or more sources about a target variable into the specific contribution of each…
This article serves as a brief introduction to the Shannon information theory. Concepts of information, Shannon entropy and channel capacity are mainly covered. All these concepts are developed in a totally combinatorial flavor. Some issues…
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of…
We derive a well-defined renormalized version of mutual information that allows to estimate the dependence between continuous random variables in the important case when one is deterministically dependent on the other. This is the situation…
In this paper we formalize the notions of information elements and information lattices, first proposed by Shannon. Exploiting this formalization, we identify a comprehensive parallelism between information lattices and subgroup lattices.…
Notwithstanding various attempts to construct a Partial Information Decomposition (PID) for multiple variables by defining synergistic, redundant, and unique information, there is no consensus on how one ought to precisely define either of…
The data for many classification problems, such as pattern and speech recognition, follow mixture distributions. To quantify the optimum performance for classification tasks, the Shannon mutual information is a natural information-theoretic…
How can the information that a set ${X_{1},...,X_{n}}$ of random variables contains about another random variable $S$ be decomposed? To what extent do different subgroups provide the same, i.e. shared or redundant, information, carry unique…
This paper examines how an event from one random variable provides pointwise mutual information about an event from another variable via probability mass exclusions. We start by introducing probability mass diagrams, which provide a visual…
We derive the property of strong superadditivity of mutual information arising from the Markov property of the vacuum state in a conformal field theory and strong subadditivity of entanglement entropy. We show this inequality encodes…
Shannon information has, in the past, been applied to quantify the genetic diversity of many natural populations. Here, we apply the Shannon concept to consecutive generations of alleles as they evolve over time. We suppose a genetic system…
In network science, researchers often use mutual information to understand the difference between network partitions produced by community detection methods. Here we extend the use of mutual information to covers, that is, the cases where a…
Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as…
We derive the closed-form expression of the maximum mutual information - the maximum value of $I(X;Z)$ obtainable via training - for a broad family of neural network architectures. The quantity is essential to several branches of machine…
To fully characterize the information that two `source' variables carry about a third `target' variable, one must decompose the total information into redundant, unique and synergistic components, i.e. obtain a partial information…
Information is the basic concept of information theory. However, there is no definition of this concept that can encompass all uses of the term information in information theories and beyond. Many question a possibility of such a…