Related papers: Information Theoretic Bounds on Optimal Worst-case…
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output…
Learning from noisy labels (LNL) is crucial in deep learning, in which one of the approaches is to identify clean-label samples from poorly-annotated datasets. Such an identification is challenging because the conventional LNL problem,…
Motivated by recent studies of population coding in theoretical neuroscience, we examine the optimality of a recently described form of stochastic resonance known as suprathreshold stochastic resonance, which occurs in populations of noisy…
The action of a noise operator on a code transforms it into a distribution on the respective space. Some common examples from information theory include Bernoulli noise acting on a code in the Hamming space and Gaussian noise acting on a…
Hypothesis testing is a fundamental issue in statistical inference and has been a crucial element in the development of information sciences. The Chernoff bound gives the minimal Bayesian error probability when discriminating two hypotheses…
We propose a new analysis framework for clustering $M$ items into an unknown number of $K$ distinct groups using noisy and actively collected responses. At each time step, an agent is allowed to query pairs of items and observe bandit…
Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify…
Bayesian Neural Networks with Latent Variables (BNN+LVs) capture predictive uncertainty by explicitly modeling model uncertainty (via priors on network weights) and environmental stochasticity (via a latent input noise variable). In this…
In this paper we derive information theoretic performance bounds to sensing and reconstruction of sparse phenomena from noisy projections. We consider two settings: output noise models where the noise enters after the projection and input…
A novel method for estimating Bayesian network (BN) parameters from data is presented which provides improved performance on test data. Previous research has shown the value of representing conditional probability distributions (CPDs) via…
Assuming that data are collected sequentially from independent streams, we consider the simultaneous testing of multiple binary hypotheses under two general setups; when the number of signals (correct alternatives) is known in advance, and…
Statistically consistent methods based on the noise transition matrix ($T$) offer a theoretically grounded solution to Learning with Noisy Labels (LNL), with guarantees of convergence to the optimal clean-data classifier. In practice,…
We study a distributed hypothesis testing setup where peripheral nodes send quantized data to the fusion center in a memoryless fashion. The \emph{expected} number of bits sent by each node under the null hypothesis is kept limited. We…
Information theory is introduced in this lecture note with a particular emphasis on its relevance to algebraic coding theory. The document develops the mathematical foundations for quantifying uncertainty and information transmission by…
Tight lower and upper bounds on the ratio of relative entropies of two probability distributions with respect to a common third one are established, where the three distributions are collinear in the standard $(n-1)$-simplex. These bounds…
We consider symmetric hypothesis testing in quantum statistics, where the hypotheses are density operators on a finite-dimensional complex Hilbert space, representing states of a finite quantum system. We prove a lower bound on the…
This work considers the problem of the noisy binary search in a sorted array. The noise is modeled by a parameter $p$ that dictates that a comparison can be incorrect with probability $p$, independently of other queries. We state two types…
We consider codes for channels with extreme noise that emerge in various low-power applications. Simple LDPC-type codes with parity checks of weight 3 are first studied for any dimension $m\rightarrow\infty.$ These codes form modulation…
The muti-layer information bottleneck (IB) problem, where information is propagated (or successively refined) from layer to layer, is considered. Based on information forwarded by the preceding layer, each stage of the network is required…
We consider the problem of identifying the most influential nodes for a spreading process on a network when prior knowledge about structure and dynamics of the system is incomplete or erroneous. Specifically, we perform a numerical analysis…