Related papers: Nonlinear Information Theory: Characterizing Distr…
We present novel data-processing inequalities relating the mutual information and the directed information in systems with feedback. The internal blocks within such systems are restricted only to be causal mappings, but are allowed to be…
We define the concept of dependence among multiple variables using maximum entropy techniques and introduce a graphical notation to denote the dependencies. Direct inference of information theoretic quantities from data uncovers…
In the era of big data, it is necessary to split extremely large data sets across multiple computing nodes and construct estimators using the distributed data. When designing distributed estimators, it is desirable to minimize the amount of…
Information Theory concepts and methodologies conform the background of how communication systems are studied and understood. They are mainly focused on the source-channel-receiver problem and on the asymptotic limits of accuracy and…
Notwithstanding various attempts to construct a Partial Information Decomposition (PID) for multiple variables by defining synergistic, redundant, and unique information, there is no consensus on how one ought to precisely define either of…
Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as…
From the output produced by a memoryless deletion channel with a uniformly random input of known length $n$, one obtains a posterior distribution on the channel input. The difference between the Shannon entropy of this distribution and that…
This paper deals with uncertainty quantification and out-of-distribution detection in deep learning using Bayesian and ensemble methods. It proposes a practical solution to the lack of prediction diversity observed recently for standard…
We develop a new approach to estimate the uncertainty due to missing higher orders in perturbative predictions (the perturbative "theory uncertainty"), which overcomes many inherent limitations of the currently prevalent methods based on…
We consider the problem of communication over a network containing a hidden and malicious adversary that can control a subset of network resources, and aims to disrupt communications. We focus on omniscient node-based adversaries, i.e., the…
We characterize information as risk reduction between knowledge states represented by partitions of the underlying probability space. Entropy corresponds to risk reduction from no (or partial) knowledge to full knowledge about a random…
In this paper, we consider a recently-proposed model of teaching and learning under uncertainty, in which a teacher receives independent observations of a single bit corrupted by binary symmetric noise, and sequentially transmits to a…
We evaluate the mutual information between the input and the output of a two layer network in the case of a noisy and non-linear analogue channel. In the case where the non-linearity is small with respect to the variability in the noise, we…
Neural networks predictions are unreliable when the input sample is out of the training distribution or corrupted by noise. Being able to detect such failures automatically is fundamental to integrate deep learning algorithms into robotics.…
The increased uncertainty and complexity of nonlinear systems have motivated investigators to consider generalized approaches to defining an entropy function. New insights are achieved by defining the average uncertainty in the probability…
We study the following one-way asymmetric transmission problem, also a variant of model-based compressed sensing: a resource-limited encoder has to report a small set $S$ from a universe of $N$ items to a more powerful decoder (server). The…
We investigate the unsupervised node classification problem on random hypergraphs under the non-uniform Hypergraph Stochastic Block Model (HSBM) with two equal-sized communities. In this model, edges appear independently with probabilities…
A joint communication and channel state estimation problem is investigated, in which reliable information transmission over a noisy channel, and high-fidelity estimation of the channel state, are simultaneously sought. The tradeoff between…
Shannon Entropy is the preeminent tool for measuring the level of uncertainty (and conversely, information content) in a random variable. In the field of communications, entropy can be used to express the information content of given…
In this article, we establish a comprehensive theoretical framework for remote estimation in a networked system composed of a source that is observed by a sensor, a remote monitor that needs to estimate the state of the source in real time,…