Related papers: Information Theory Measures via Multidimensional G…
We try to establish a unified information theoretic approach to learning and to explore some of its applications. First, we define {\em predictive information} as the mutual information between the past and the future of a time series,…
Recent advances in neuroscientific experimental techniques have enabled us to simultaneously record the activity of thousands of neurons across multiple brain regions. This has led to a growing need for computational tools capable of…
We introduce a hierarchical classification of theories that describe systems with fundamentally limited information content. This property is introduced in an operational way and gives rise to the existence of mutually complementary…
Training Data Attribution (TDA) seeks to trace model predictions back to influential training examples, enhancing interpretability and safety. We formulate TDA as a Bayesian information-theoretic problem: subsets are scored by the…
We define a measure of redundant information based on projections in the space of probability distributions. Redundant information between random variables is information that is shared between those variables. But in contrast to mutual…
How much visual information about the retinal images can be extracted from the different layers of the visual pathway?. Separate subsystems (e.g. opponent channels, spatial filters, nonlinearities of the texture sensors) have been suggested…
Observational entropy is interpreted as the uncertainty an observer making measurements associates with a system. So far, properties that make such an interpretation possible rely on the assumption of ideal projective measurements. We show…
Although information theory resolves inconsistencies (known under the form of famous enigmas) of the traditional approach of thermostatistics, its place in the corresponding literature is not what it deserves. This article supports the idea…
Information theory provides principled ways to analyze different inference and learning problems such as hypothesis testing, clustering, dimensionality reduction, classification, among others. However, the use of information theoretic…
We are interested in learning data-driven representations that can generalize well, even when trained on inherently biased data. In particular, we face the case where some attributes (bias) of the data, if learned by the model, can severely…
Gaussian process regression is widely used because of its ability to provide well-calibrated uncertainty estimates and handle small or sparse datasets. However, it struggles with high-dimensional data. One possible way to scale this…
One of the fundamental steps toward understanding a complex system is identifying variation at the scale of the system's components that is most relevant to behavior on a macroscopic scale. Mutual information provides a natural means of…
Our understanding of complex systems rests on our ability to characterise how they perform distributed computation and integrate information. Advances in information theory have introduced several quantities to describe complex information…
This paper evaluates heterogeneous information fusion using multi-task Gaussian processes in the context of geological resource modeling. Specifically, it empirically demonstrates that information integration across heterogeneous…
Across the scientific literature, information measurement in the nervous system is posed as a problem of information processing internal to the brain by constructs such as neuronal populations, sensory surprise, or cognitive models.…
Formalising the confrontation of opinions (models) to observations (data) is the task of Inferential Statistics. Information Theory provides us with a basic functional, the relative entropy (or Kullback-Leibler divergence), an asymmetrical…
Missing data theory deals with the statistical methods in the occurrence of missing data. Missing data occurs when some values are not stored or observed for variables of interest. However, most of the statistical theory assumes that data…
This paper applies the recently axiomatized Optimum Information Principle (minimize the Kullback-Leibler information subject to all relevant information) to nonparametric density estimation, which provides a theoretical foundation as well…
Multivariate mutual information provides a conceptual framework for characterizing higher-order interactions in complex systems. Two well-known measures of multivariate information---total correlation and dual total correlation---admit a…
Importance sampling approximates expectations with respect to a target measure by using samples from a proposal measure. The performance of the method over large classes of test functions depends heavily on the closeness between both…