Related papers: I-MMSE relations in random linear estimation and a…
In a previous report we have evaluated analytically the mutual information between the firing rates of N independent units and a set of continuous+discrete stimuli, for finite N and in the limit of large noise. Here, we extend the analysis…
Among the many ways to model signals, a recent approach that draws considerable attention is sparse representation modeling. In this model, the signal is assumed to be generated as a random linear combination of a few atoms from a…
We consider distributed recursive estimation of consensus+innovations type in the presence of heavy-tailed sensing and communication noises. We allow that the sensing and communication noises are mutually correlated while independent…
Line spectral estimation (LSE) from multi snapshot samples is studied utilizing the variational Bayesian methods. Motivated by the recently proposed variational line spectral estimation (VALSE) method for a single snapshot, we develop the…
Information theory is built on probability measures and by definition a probability measure has total mass 1. Probability measures are used to model uncertainty, and one may ask how important it is that the total mass is one. We claim that…
Threshold and ambiguity phenomena are studied in Part 1 of this work where approximations for the mean-squared-error (MSE) of the maximum likelihood estimator are proposed using the method of interval estimation (MIE), and where approximate…
We examine a class of deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical…
The minimum mean square error of the estimation of a non Gaussian signal where observed from an additive white Gaussian noise channel's output, is analyzed. First, a quite general time-continuous channel model is assumed for which the…
Mutual information is fundamentally important for measuring statistical dependence between variables and for quantifying information transfer by signaling and communication mechanisms. It can, however, be challenging to evaluate for…
The ability to understand and solve high-dimensional inference problems is essential for modern data science. This article examines high-dimensional inference problems through the lens of information theory and focuses on the standard…
Mutual information (MI) is a fundamental quantity in information theory and machine learning. However, direct estimation of MI is intractable, even if the true joint probability density for the variables of interest is known, as it involves…
In diffusion-based communication, as for molecular systems, the achievable data rate is low due to the stochastic nature of diffusion which exhibits a severe inter-symbol-interference (ISI). Multiple-Input Multiple-Output (MIMO)…
An approximate mean square error (MSE) expression for the performance analysis of implicitly defined estimators of non-random parameters is proposed. An implicitly defined estimator (IDE) declares the minimizer/maximizer of a selected…
Implicit samplers are algorithms for producing independent, weighted samples from multi-variate probability distributions. These are often applied in Bayesian data assimilation algorithms. We use Laplace asymptotic expansions to analyze two…
Sensing performance is typically evaluated by classical radar metrics, such as Cramer-Rao bound and signal-to-clutter-plus-noise ratio. The recent development of the integrated sensing and communication (ISAC) framework motivated the…
Mutual information (MI) is a useful information-theoretic measure to quantify the statistical dependence between two random variables: $X$ and $Y$. Often, we are interested in understanding how the dependence between $X$ and $Y$ in one set…
As one of the central tasks in machine learning, regression finds lots of applications in different fields. An existing common practice for solving regression problems is the mean square error (MSE) minimization approach or its regularized…
We present new fundamental results for the mean square error (MSE)-optimal conditional mean estimator (CME) in one-bit quantized systems for a Gaussian mixture model (GMM) distributed signal of interest, possibly corrupted by additive white…
A continuing mystery in understanding the empirical success of deep neural networks is their ability to achieve zero training error and generalize well, even when the training data is noisy and there are more parameters than data points. We…
The data for many classification problems, such as pattern and speech recognition, follow mixture distributions. To quantify the optimum performance for classification tasks, the Shannon mutual information is a natural information-theoretic…