Related papers: A Separation between Divergence and Holevo Informa…
This paper develops a new divergence that generalizes relative entropy and can be used to compare probability measures without a requirement of absolute continuity. We establish properties of the divergence, and in particular derive and…
In multicenter research, individual-level data are often protected against sharing across sites. To overcome the barrier of data sharing, many distributed algorithms, which only require sharing aggregated information, have been developed.…
Change of measure inequalities translate divergences between probability measures into explicit bounds on event probabilities, and play an important role in deriving probabilistic guarantees in learning theory, information theory, and…
Common misconceptions on the Heisenberg principle are reviewed, and the original spirit of the principle is reestablished in terms of the trade-off between information retrieved by a measurement and disturbance on the measured system. After…
There are many information and divergence measures exist in the literature on information theory and statistics. The most famous among them are Kullback-Leiber's (1951)relative information and Jeffreys (1946) J-divergence, Information…
There are many information and divergence measures exist in the literature on information theory and statistics. The most famous among them are Kullback-Leibler (1951) relative information and Jeffreys (1951) J-divergence. Sibson (1969)…
This paper is an attempt to set a justification for making use of some dicrepancy indexes, starting from the classical Maximum Likelihood definition, and adapting the corresponding basic principle of inference to situations where…
Message identification (M-I) divergence is an important measure of the information distance between probability distributions, similar to Kullback-Leibler (K-L) and Renyi divergence. In fact, M-I divergence with a variable parameter can…
Estimating statistical models within sensor networks requires distributed algorithms, in which both data and computation are distributed across the nodes of the network. We propose a general approach for distributed learning based on…
We show that the $k$th order statistic from a heterogeneous sample of $n\geq k$ exponential random variables is larger than that from a homogeneous exponential sample in the sense of star ordering, as conjectured by Xu and Balakrishnan…
Mutual information $I(X;Y)$ is a useful definition in information theory to estimate how much information the random variable $Y$ holds about the random variable $X$. One way to define the mutual information is by comparing the joint…
The distcomp command is introduced and illustrated. The command assesses whether or not two distributions differ at each possible value while controlling the probability of any false positive, even in finite samples. Syntax and the…
Real-life data are often non-IID due to complex distributions and interactions, and the sensitivity to the distribution of samples can differ among learning models. Accordingly, a key question for any supervised or unsupervised model is…
There are three classical divergence measures exist in the literature on information theory and statistics. These are namely, Jeffryes-Kullback-Leiber J-divergence. Sibson-Burbea-Rao Jensen-Shannon divegernce and Taneja arithemtic-geometric…
How much one has learned from an experiment is quantifiable by the information gain, also known as the Kullback-Leibler divergence. The narrowing of the posterior parameter distribution $P(\theta|D)$ compared with the prior parameter…
Contrastive learning has emerged as a cornerstone in recent achievements of unsupervised representation learning. Its primary paradigm involves an instance discrimination task with a mutual information loss. The loss is known as InfoNCE and…
Two new information-theoretic methods are introduced for establishing Poisson approximation inequalities. First, using only elementary information-theoretic techniques it is shown that, when $S_n=\sum_{i=1}^nX_i$ is the sum of the (possibly…
Pimentel et al. (2020) recently analysed probing from an information-theoretic perspective. They argue that probing should be seen as approximating a mutual information. This led to the rather unintuitive conclusion that representations…
Information measures can be constructed from R\'enyi divergences much like mutual information from Kullback-Leibler divergence. One such information measure is known as Sibson $\alpha$-mutual information and has received renewed attention…
The Holevo bound is a bound on the mutual information for a given quantum encoding. In 1996 Schumacher, Westmoreland and Wootters [Schumacher, Westmoreland and Wootters, Phys. Rev. Lett. 76, 3452 (1996)] derived a bound which reduces to the…