Related papers: Mutual Information and Conditional Mean Prediction…

While the linear Pearson correlation coefficient represents a well-established normalized measure to quantify the interrelation of two stochastic variables $X$ and $Y$, it fails for multidimensional variables such as Cartesian coordinates.…

The use of Mutual Information (MI) as a measure to evaluate the efficiency of cryptosystems has an extensive history. However, estimating MI between unknown random variables in a high-dimensional space is challenging. Recent advances in…

Most work in algorithmic fairness to date has focused on discrete outcomes, such as deciding whether to grant someone a loan or not. In these classification settings, group fairness criteria such as independence, separation and sufficiency…

Most existing works on disentangled representation learning are solely built upon an marginal independence assumption: all factors in disentangled representations should be statistically independent. This assumption is necessary but…

Analysis of a probabilistic system often requires to learn the joint probability distribution of its random variables. The computation of the exact distribution is usually an exhaustive precise analysis on all executions of the system. To…

Mutual information (MI) is a fundamental quantity in information theory and machine learning. However, direct estimation of MI is intractable, even if the true joint probability density for the variables of interest is known, as it involves…

Feature selection problems arise in a variety of applications, such as microarray analysis, clinical prediction, text categorization, image classification and face recognition, multi-label learning, and classification of internet traffic.…

The Bayesian measure of sample information about the parameter, known as Lindley's measure, is widely used in various problems such as developing prior distributions, models for the likelihood functions and optimal designs. The predictive…

Measuring model performance is a key issue for deep learning practitioners. However, we often lack the ability to explain why a specific architecture attains superior predictive accuracy for a given data set. Often, validation accuracy is…

Dynamic feature selection, where we sequentially query features to make accurate predictions with a minimal budget, is a promising paradigm to reduce feature acquisition costs and provide transparency into a model's predictions. The problem…

In the setting where information cannot be verified, we propose a simple yet powerful information theoretical framework---the Mutual Information Paradigm---for information elicitation mechanisms. Our framework pays every agent a measure of…

Testing hypothesis of independence between two random elements on a joint alphabet is a fundamental exercise in statistics. Pearson's chi-squared test is an effective test for such a situation when the contingency table is relatively small.…

We discuss the connection between information and copula theories by showing that a copula can be employed to decompose the information content of a multivariate distribution into marginal and dependence components, with the latter…

On the basis of local realism theory, nonlocal information is necessary for violation of Bell's inequality. From a theoretical point of view, nonlocal information is essentially the mutual information on distant outcome and measurement…

We provide an information-theoretic framework for studying the generalization properties of machine learning algorithms. Our framework ties together existing approaches, including uniform convergence bounds and recent methods for adaptive…

The amount of mutual information contained in time series of two elements gives a measure of how well their activities are coordinated. In a large, complex network of interacting elements, such as a genetic regulatory network within a cell,…

In this paper, a lower bound of quantum conditional mutual information is obtained by employing the Peierls-Bogoliubov inequality and Golden Thompson inequality. Comparison with the bounds obtained by other researchers indicates that our…

This paper develops a new framework, called modular regression, to utilize auxiliary information -- such as variables other than the original features or additional data sets -- in the training process of linear models. At a high level, our…

Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a…

The mutual information method has demonstrated to be very useful for deriving the potential order parameter of a system. Although the method suggests some constraints which help to define this quantity, there is still some freedom in the…