Related papers: On Strong Data-Processing and Majorization Inequal…
This paper studies a Shannon-theoretic version of the generalized distribution preserving quantization problem where a stationary and memoryless source is encoded subject to a distortion constraint and the additional requirement that the…
This paper considers the joint compression of a pair of correlated sources, where the encoder is allowed to access only one of the sources. The objective is to recover both sources under separate distortion constraints for each source while…
Fano's inequality reveals the relation between the conditional entropy and the probability of error . It has been the key tool in proving the converse of coding theorems in the past sixty years. In this paper, an extended Fano's inequality…
The problem of guessing a random string is revisited. A close relation between guessing and compression is first established. Then it is shown that if the sequence of distributions of the information spectrum satisfies the large deviation…
Can we use machine learning to compress graph data? The absence of ordering in graphs poses a significant challenge to conventional compression algorithms, limiting their attainable gains as well as their ability to discover relevant…
Computing the partition function and the marginals of a global probability distribution are two important issues in any probabilistic inference problem. In a previous work, we presented sub-tree based upper and lower bounds on the partition…
Recently, information theoretic analysis has become a popular framework for understanding the generalization behavior of deep neural networks. It allows a direct analysis for stochastic gradient/Langevin descent (SGD/SGLD) learning…
As an alternative to the well-known methods of "chaining" and "bracketing" that have been developed in the study of random fields, a new method, which is based on a stochastic maximal inequality derived by using It\^o's formula and on a new…
A well-known but rarely used approach to text categorization uses conditional entropy estimates computed using data compression tools. Text affinity scores derived from compressed sizes can be used for classification and ranking tasks, but…
This work proposes a new loss function targeting classification problems, utilizing a source of information overlooked by cross entropy loss. First, we derive a series of the tightest upper and lower bounds for the probability of a random…
The quantum relative entropy is a measure of the distinguishability of two quantum states, and it is a unifying concept in quantum information theory: many information measures such as entropy, conditional entropy, mutual information, and…
We develop a unified nonparametric framework for sharp partial identification and inference on inequality indices when the data contain coarsened observations of the variable of interest. We characterize the extremal allocations for all…
We develop large sample theory for merged data from multiple sources. Main statistical issues treated in this paper are (1) the same unit potentially appears in multiple datasets from overlapping data sources, (2) duplicated items are not…
We investigate quantum R\'enyi entropic quantities, specifically those derived from 'sandwiched' divergence. This divergence is one of several proposed R\'enyi generalisations of the quantum relative entropy. We may define R\'enyi…
This study considers the unconditional smooth R\'{e}nyi entropy, the smooth conditional R\'{e}nyi entropy proposed by Kuzuoka [\emph{IEEE Trans.\ Inf.\ Theory}, vol.~66, no.~3, pp.~1674--1690, 2020], and a new quantity which we term the…
In this work, lossy distributed compression of pairs of correlated sources is considered. Conventionally, Shannon's random coding arguments -- using randomly generated unstructured codebooks whose blocklength is taken to be asymptotically…
Incomplete U-statistics have been proposed to accelerate computation. They use only a subset of the subsamples required for kernel evaluations by complete U-statistics. This paper gives a finite sample bound in the style of Bernstein's…
The recently developed matrix based Renyi's entropy enables measurement of information in data simply using the eigenspectrum of symmetric positive semi definite (PSD) matrices in reproducing kernel Hilbert space, without estimation of the…
Based on a factorization of an input covariance matrix, we define a mild generalization of an upper bound of Nikolov (2015) and Li and Xie (2020) for the NP-Hard constrained maximum-entropy sampling problem (CMESP). We demonstrate that this…
Entropy and relative or cross entropy measures are two very fundamental concepts in information theory and are also widely used for statistical inference across disciplines. The related optimization problems, in particular the maximization…