Related papers: Some new insights into information decomposition i…
Dependence is undoubtedly a central concept in statistics. Though, it proves difficult to locate in the literature a formal definition which goes beyond the self-evident 'dependence = non-independence'. This absence has allowed the term…
Most psychophysical experiments discard half the data collected. Specifically, experiments discard reaction time data, and use binary responses (e.g. yes/no) to measure performance. Here, Shannon's information theory is used to define…
Conceptually, partial information decomposition (PID) is concerned with separating the information contributions several sources hold about a certain target by decomposing the corresponding joint mutual information into contributions such…
Information and uncertainty are closely related and extensively studied concepts in a number of scientific disciplines such as communication theory, probability theory, and statistics. Increasing the information arguably reduces the…
Context dependence is central to the description of complexity. Keying on the pairwise definition of "set complexity" we use an information theory approach to formulate general measures of systems complexity. We examine the properties of…
The extraction of a physical law y=yo(x) from joint experimental data about x and y is treated. The joint, the marginal and the conditional probability density functions (PDF) are expressed by given data over an estimator whose kernel is…
Measuring dependence between two events, or equivalently between two binary random variables, amounts to expressing the dependence structure inherent in a $2\times 2$ contingency table in a real number between $-1$ and $1$. Countless such…
The problem of measuring conditional dependence between two random phenomena arises when a third one (a confounder) has a potential influence on the amount of information between them. A typical issue in this challenging problem is the…
Two families of dependence measures between random variables are introduced. They are based on the R\'enyi divergence of order $\alpha$ and the relative $\alpha$-entropy, respectively, and both dependence measures reduce to Shannon's mutual…
Structural causal models provide a formalism to express causal relations between variables of interest. Models and variables can represent a system at different levels of abstraction, whereby relations may be coarsened and refined according…
Observations on the past provide some hints about what will happen in the future, and this can be quantified using information theory. The ``predictive information'' defined in this way has connections to measures of complexity that have…
In this paper we consider the axiomatic characterization of information and certainty measures in a unified way. We present the general axiomatic system which captures the common properties of a large number of the measures previously…
Statistical modeling of physical laws connects experiments with mathematical descriptions of natural phenomena. The modeling is based on the probability density of measured variables expressed by experimental data via a kernel estimator. As…
In the context of statistical learning, the Information Bottleneck method seeks a right balance between accuracy and generalization capability through a suitable tradeoff between compression complexity, measured by minimum description…
This paper introduces time into information theory, gives a more accurate definition of information, and unifies the information in cognition and Shannon information theory. Specially, we consider time as a measure of information, giving a…
We consider the problem of identifying universal low-dimensional features from high-dimensional data for inference tasks in settings involving learning. For such problems, we introduce natural notions of universality and we show a local…
The study of inter-human communication requires a more complex framework than Shannon's (1948) mathematical theory of communication because "information" is defined in the latter case as meaningless uncertainty. Assuming that meaning cannot…
A core feature of complex systems is that the interactions between elements in the present causally constrain each-other as the system evolves through time. To fully model all of these interactions (between elements, as well as ensembles of…
We explore a few common models on how correlations affect information. The main model considered is the Shannon mutual information $I(S:R_1,\cdots, R_i)$ over distributions with marginals $P_{S,R_i}$ fixed for each $i$, with the analogy in…
We introduce a bottleneck method for learning data representations based on information deficiency, rather than the more traditional information sufficiency. A variational upper bound allows us to implement this method efficiently. The…