Related papers: On Strong Data-Processing and Majorization Inequal…
The problem of high-dimensional and large-scale representation of visual data is addressed from an unsupervised learning perspective. The emphasis is put on discrete representations, where the description length can be measured in bits and…
Information theory provides tools to predict the performance of a learning algorithm on a given dataset. For instance, the accuracy of learning an unknown parameter can be upper bounded by reducing the learning task to hypothesis testing…
The optimized quantum $f$-divergences form a family of distinguishability measures that includes the quantum relative entropy and the sandwiched R\'enyi relative quasi-entropy as special cases. In this paper, we establish physically…
This paper establishes a close relationship among the four information theoretic problems, namely Campbell source coding, Arikan guessing, Huleihel et al. memoryless guessing and Bunte and Lapidoth tasks partitioning problems. We first show…
In many areas of engineering and sciences, decision rules and control strategies are usually designed based on nominal values of relevant system parameters. To ensure that a control strategy or decision rule will work properly when the…
We consider list versions of sparse approximation problems, where unlike the existing results in sparse approximation that consider situations with unique solutions, we are interested in multiple solutions. We introduce these problems and…
This paper considers the problem of variable-length lossy source coding. The performance criteria are the excess distortion probability and the cumulant generating function of codeword lengths. We derive a non-asymptotic fundamental limit…
Purely multiplicative comparisons of quantum relative entropy are desirable but challenging to prove. We show such comparisons for relative entropies between comparable densities, including the relative entropy of a density with respect to…
The data-processing inequality, that is, $I(U;Y) \le I(U;X)$ for a Markov chain $U \to X \to Y$, has been the method of choice for proving impossibility (converse) results in information theory and many other disciplines. Various…
We study the statistical complexity of estimating partition functions given sample access to a proposal distribution and an unnormalized density ratio for a target distribution. While partition function estimation is a classical problem,…
Tensors are a fundamental operation in distributed computing, \emph{e.g.,} machine learning, that are commonly distributed into multiple parallel tasks for large datasets. Stragglers and other failures can severely impact the overall…
This paper proposes a unifying variational approach for proving and extending some fundamental information theoretic inequalities. Fundamental information theory results such as maximization of differential entropy, minimization of Fisher…
Data compression is a well-studied (and well-solved) problem in the setup of long coding blocks. But important emerging applications need to compress data to memory words of small fixed widths. This new setup is the subject of this paper.…
This paper starts by considering the minimization of the Renyi divergence subject to a constraint on the total variation distance. Based on the solution of this optimization problem, the exact locus of the points $\bigl( D(Q\|P_1),…
In many high-impact applications, it is important to ensure the quality of output of a machine learning algorithm as well as its reliability in comparison with the complexity of the algorithm used. In this paper, we have initiated a…
Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to…
A deep learning system typically suffers from a lack of reproducibility that is partially rooted in hardware or software implementation details. The irreproducibility leads to skepticism in deep learning technologies and it can hinder them…
Let ($X,Y)$ be a random vector with distribution function $F(x,y),$ and $(X_{1},Y_{1}),(X_{2},Y_{2}),...,(X_{n},Y_{n})$ are independent copies of ($X,Y).$ Let $X_{i:n}$ be the $i$th order statistics constructed from the sample…
In filtering, each output is produced by a certain number of different inputs. We explore the statistics of this degeneracy in an explicitly treatable filtering problem in which filtering performs the maximal compression of relevant…
Stimulated by the need of describing useful notions related to information measures, we introduce the `pdf-related distributions'. These are defined in terms of transformation of absolutely continuous random variables through their own…