Related papers: Distribution of Mutual Information
Bayesian inference promises a framework for principled uncertainty quantification of neural network predictions. Barriers to adoption include the difficulty of fully characterizing posterior distributions on network parameters and the…
The ability of information processing in biologically motivated Boolean networks is of interest in recent information theoretic research. One measure to quantify this ability is the well known mutual information. Using Fourier analysis we…
We are interested in learning data-driven representations that can generalize well, even when trained on inherently biased data. In particular, we face the case where some attributes (bias) of the data, if learned by the model, can severely…
Analysis of a probabilistic system often requires to learn the joint probability distribution of its random variables. The computation of the exact distribution is usually an exhaustive precise analysis on all executions of the system. To…
Organisms and algorithms learn probability distributions from previous observations, either over evolutionary time or on the fly. In the absence of regularities, estimating the underlying distribution from data would require observing each…
One of the well-known challenges in optimal experimental design is how to efficiently estimate the nested integrations of the expected information gain. The Gaussian approximation and associated importance sampling have been shown to be…
Machine learning theory has mostly focused on generalization to samples from the same distribution as the training data. Whereas a better understanding of generalization beyond the training distribution where the observed distribution…
The distribution of the sum of independent identically distributed uniform random variables is well-known. However, it is sometimes necessary to analyze data which have been drawn from different uniform distributions. By inverting the…
Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order, in which the retrieved documents are presented to the model when solving tasks such as question answering (QA).…
Following the idea of Bayesian learning via Gaussian mixture model, we organically combine the backward-looking information contained in the historical data and the forward-looking information implied by the market portfolio, which is…
Diffusion bridge models in both continuous and discrete state spaces have recently become powerful tools in the field of generative modeling. In this work, we leverage the discrete state space formulation of bridge matching models to…
Mutual information is a nonlinear measure used in time series analysis in order to measure the linear and non-linear correlations at any lag $\tau$. The aim of this study is to evaluate some of the most commonly used mutual information…
The estimation of mutual information (MI) or conditional mutual information (CMI) from a set of samples is a long-standing problem. A recent line of work in this area has leveraged the approximation power of artificial neural networks and…
In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to…
The use of Mutual Information (MI) as a measure to evaluate the efficiency of cryptosystems has an extensive history. However, estimating MI between unknown random variables in a high-dimensional space is challenging. Recent advances in…
One of the most complex tasks of decision making and planning is to gather information. This task becomes even more complex when the state is high-dimensional and its belief cannot be expressed with a parametric distribution. Although the…
Information estimates such as the ``direct method'' of Strong et al. (1998) sidestep the difficult problem of estimating the joint distribution of response and stimulus by instead estimating the difference between the marginal and…
In Generalised Bayesian Inference (GBI), the learning rate and hyperparameters of the loss must be estimated. These inference-hyperparameters can't be estimated jointly with the other parameters, from the data, by giving them a prior.…
Between Bayesian and frequentist inference, it's commonly believed that the former is for cases where one has a prior and the latter is for cases where one has no prior. But the prior/no-prior classification isn't exhaustive, and most…
The Bayesian measure of sample information about the parameter, known as Lindley's measure, is widely used in various problems such as developing prior distributions, models for the likelihood functions and optimal designs. The predictive…