Related papers: Two Measures of Dependence
Measuring the dependence of data plays a central role in statistics and machine learning. In this work, we summarize and generalize the main idea of existing information-theoretic dependence measures into a higher-level perspective by the…
The R\'enyi information measures are characterized in terms of their Shannon counterparts, and properties of the former are recovered from first principle via the associated properties of the latter. Motivated by this characterization, a…
The achievable error-exponent pairs for the type I and type II errors are characterized in a hypothesis testing setup where the observation consists of independent and identically distributed samples from either a known joint probability…
Algorithmic entropy and Shannon entropy are two conceptually different information measures, as the former is based on size of programs and the later in probability distributions. However, it is known that, for any recursive probability…
In this article, we discuss the problem of establishing relations between information measures assessed for network structures. Two types of entropy based measures namely, the Shannon entropy and its generalization, the R\'{e}nyi entropy…
We explore a family of information measures that stems from R\'enyi's $\alpha$-Divergences with $\alpha<0$. In particular, we extend the definition of Sibson's $\alpha$-Mutual Information to negative values of $\alpha$ and show several…
We calculate and analyze various entropy measures and their properties for selected probability distributions. The entropies considered include Shannon, R\'enyi, generalized R\'enyi, Tsallis, Sharma-Mittal, and modified Shannon entropy,…
Entropy is a measure of self-information which is used to quantify losses. Entropy was developed in thermodynamics, but is also used to compare probabilities based on their deviating information content. Corresponding model uncertainty is…
Measuring dependence between two events, or equivalently between two binary random variables, amounts to expressing the dependence structure inherent in a $2\times 2$ contingency table in a real number between $-1$ and $1$. Countless such…
Consider a random sample $X_1 , X_2 , ..., X_n$ drawn independently and identically distributed from some known sampling distribution $P_X$. Let $X_{(1)} \le X_{(2)} \le ... \le X_{(n)}$ represent the order statistics of the sample. The…
The data for many classification problems, such as pattern and speech recognition, follow mixture distributions. To quantify the optimum performance for classification tasks, the Shannon mutual information is a natural information-theoretic…
Accurately determining dependency structure is critical to discovering a system's causal organization. We recently showed that the transfer entropy fails in a key aspect of this---measuring information flow---due to its conflation of dyadic…
The paper presents a new copula based method for measuring dependence between random variables. Our approach extends the Maximum Mean Discrepancy to the copula of the joint distribution. We prove that this approach has several advantageous…
We introduce a family of information leakage measures called maximal $\alpha,\beta$-leakage, parameterized by real numbers $\alpha$ and $\beta$. The measure is formalized via an operational definition involving an adversary guessing an…
Information theoretic measures (entropies, entropy rates, mutual information) are nowadays commonly used in statistical signal processing for real-world data analysis. The present work proposes the use of Auto Mutual Information (Mutual…
Higher-order information theory has become a rapidly growing toolkit in computational neuroscience, motivated by the idea that multivariate dependencies can reveal aspects of neural computation and communication that are invisible to…
R\'enyi divergence is related to R\'enyi entropy much like information divergence (also called Kullback-Leibler divergence or relative entropy) is related to Shannon's entropy, and comes up in many settings. It was introduced by R\'enyi as…
The weak law of large numbers implies that, under mild assumptions on the source, the Renyi entropy per produced symbol converges (in probability) towards the Shannon entropy rate. This paper quantifies the speed of this convergence for…
We discuss the construction of component importance measures for binary coherent reliability systems from known stochastic dependence measures by measuring the dependence between system and component failures. We treat both the…
It is not obvious how to extend Shannon's original information entropy to higher dimensions, and many different approaches have been tried. We replace the English text symbol sequence originally used to illustrate the theory by a discrete,…