Related papers: Conditional Mutual Information Neural Estimator

Learning good representations is of crucial importance in deep learning. Mutual Information (MI) or similar measures of statistical dependence are promising tools for learning these representations in an unsupervised way. Even though the…

In this paper, a lower bound of quantum conditional mutual information is obtained by employing the Peierls-Bogoliubov inequality and Golden Thompson inequality. Comparison with the bounds obtained by other researchers indicates that our…

Estimation of individual treatment effects is commonly used as the basis for contextual decision making in fields such as healthcare, education, and economics. However, it is often sufficient for the decision maker to have estimates of…

Results for estimating the convergence rate of non-stationary distributed consensus algorithms are provided, on the basis of qualitative (mainly topological) as well as basic quantitative information (lower-bounds on the matrix entries).…

In this chapter, we present our recent invention, i.e., the notion of the value of information$\unicode{x2014}$a semantic metric that is fundamental for networked control systems tasks. We begin our analysis by formulating a causal tradeoff…

We consider a basic joint communication and sensing setup comprising a transmitter, a receiver and a sensor. The transmitter sends a codeword to the receiver through a discrete memoryless channel, and the receiver is interested in decoding…

In this paper, we study a model of communication under adversarial noise. In this model, the adversary makes online decisions on whether to corrupt a transmitted bit based on only the value of that bit. Like the usual binary symmetric…

Deep learning systems have been reported to achieve state-of-the-art performances in many applications, and a key is the existence of well trained classifiers on benchmark datasets. As a main-stream loss function, the cross entropy can…

Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a…

A communication setup is considered where a transmitter wishes to simultaneously sense its channel state and convey a message to a receiver. The state is estimated at the transmitter by means of generalized feedback, i.e. a strictly causal…

In this paper, a simple memory limited transmitter for molecular communication is proposed, in which information is encoded in the diffusion rate of the molecules. Taking advantage of memory, the proposed transmitter reduces the ISI problem…

We present a new family of information-theoretic generalization bounds within the framework of conditional mutual information (CMI). Most of our results are established based on the leave-$m$-out (L$m$O) cross-validation error, with $m$…

We introduce Neural Conditional Probability (NCP), an operator-theoretic approach to learning conditional distributions with a focus on statistical inference tasks. NCP can be used to build conditional confidence regions and extract key…

Resource-constrained systems are prevalent in communications. Such a system is composed of many components but only some of them can be allocated with resources such as time slots. According to the amount of information about the system,…

Mutual Information (MI) is an useful tool for the recognition of mutual dependence berween data sets. Differen methods for the estimation of MI have been developed when both data sets are discrete or when both data sets are continuous. The…

Estimating mutual information (MI) between two continuous random variables $X$ and $Y$ allows to capture non-linear dependencies between them, non-parametrically. As such, MI estimation lies at the core of many data science applications.…

In multi-task adversarial networks, the accurate estimation of unknown parameters in a distributed algorithm is hindered by attacked nodes or links. To tackle this challenge, this brief proposes a low-communication resilient distributed…

We consider the problem of distributed Kalman filtering for sensor networks in the case there is a limit in data transmission and there is model uncertainty. More precisely, we propose a distributed filtering strategy with event-triggered…

Multimodal large language models (MLLMs) have shown promising capabilities but struggle under distribution shifts, where evaluation data differ from instruction tuning distributions. Although previous works have provided empirical…

We propose a novel estimator of the mutual information between two ordinal vectors $x$ and $y$. Our approach is inductive (as opposed to deductive) in that it depends on the data generating distribution solely through some nonparametric…