Related papers: Synergy, suppression and immorality: forward diffe…

Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated…

We provide a mutual information lower bound that can be used to analyze the effect of training in models with unknown parameters. For large-scale systems, we show that this bound can be calculated using the difference between two…

Complex systems are characterised by a tight, nontrivial interplay of their constituents, which gives rise to a multi-scale spectrum of emergent properties. In this scenario, it is practically and conceptually difficult to identify those…

Entropy is a measure of self-information which is used to quantify losses. Entropy was developed in thermodynamics, but is also used to compare probabilities based on their deviating information content. Corresponding model uncertainty is…

We use a well known model (T. Vicsek et al. Phys Rev Lett 15, 1226 (1995)) for flocking to test mutual information as a tool for detecting order-disorder transitions, in particular when observations of the system are limited. We show that…

We prove a lower bound on the relative entropy between two finite-dimensional states in terms of their entropy difference and the dimension of the underlying space. The inequality is tight in the sense that equality can be attained for any…

While mutual information effectively quantifies dependence between two variables, it does not by itself reveal the complex, fine-grained interactions among variables, i.e., how multiple sources contribute redundantly, uniquely, or…

Fields like public health, public policy, and social science often want to quantify the degree of dependence between variables whose relationships take on unknown functional forms. Typically, in fact, researchers in these fields are…

A new expression as a certain asymptotic limit via "discrete micro-states" of permutations is provided to the mutual information of both continuous and discrete random variables.

We characterize information as risk reduction between knowledge states represented by partitions of the underlying probability space. Entropy corresponds to risk reduction from no (or partial) knowledge to full knowledge about a random…

Our purpose is to model the dependence between two random variables, taking into account a priori knowledge on these variables. For example, in many applications (oceanography, finance...), there exists an order relation between the two…

We consider some formulations of the entropy bounds at the semiclassical level. The entropy S(V) localized in a region V is divergent in quantum field theory (QFT). Instead of it we focus on the mutual information I(V,W)=S(V)+S(W)-S(V\cup…

We investigate the possibility of distinguishing among different causal relations starting from a limited set of marginals. Our main tool is the notion of adhesivity, that is, the extension of probability or entropies defined only on…

Expectation-Maximization (EM) is a prominent approach for parameter estimation of hidden (aka latent) variable models. Given the full batch of data, EM forms an upper-bound of the negative log-likelihood of the model at each iteration and…

In a system of three stochastic variables, the Partial Information Decomposition (PID) of Williams and Beer dissects the information that two variables (sources) carry about a third variable (target) into nonnegative information atoms that…

One of the crucial steps in scientific studies is to specify dependent relationships among factors in a system of interest. Given little knowledge of a system, can we characterize the underlying dependent relationships through observation…

Bayesian inference paradigms are regarded as powerful tools for solution of inverse problems. However, when applied to inverse problems in physical sciences, Bayesian formulations suffer from a number of inconsistencies that are often…

The problem of determining the best achievable performance of arbitrary lossless compression algorithms is examined, when correlated side information is available at both the encoder and decoder. For arbitrary source-side information pairs,…

We correct claims about lower bounds on mutual information (MI) between real-valued random variables made in A. Kraskov {\it et al.}, Phys. Rev. E {\bf 69}, 066138 (2004). We show that non-trivial lower bounds on MI in terms of linear…

Interval-censored competing risks data arise when each study subject may experience an event or failure from one of several causes and the failure time is not observed exactly but rather known to lie in an interval between two successive…