Related papers: R\'{e}nyi Cross-Entropy Measures for Common Distri…
We study the $p$-R\'{e}nyi entropy power inequality with a weight factor $t$ on two independent continuous random variables $X$ and $Y$. The extension essentially relies on a modulation on the sharp Young's inequality due to Bobkov and…
We consider the entropy of sums of independent discrete random variables, in analogy with Shannon's Entropy Power Inequality, where equality holds for normals. In our case, infinite divisibility suggests that equality should hold for…
Quantum key distribution requires tight and reliable bounds on the secret key rate to ensure robust security. This is particularly so for the regime of finite block sizes, where the optimization of generalized R\'enyi entropic quantities is…
Although the recent progress is substantial, deep learning methods can be vulnerable to the maliciously generated adversarial examples. In this paper, we present a novel training procedure and a thresholding test strategy, towards robust…
Modern neural architectures for classification tasks are trained using the cross-entropy loss, which is widely believed to be empirically superior to the square loss. In this work we provide evidence indicating that this belief may not be…
Cross-entropy loss and focal loss are the most common choices when training deep neural networks for classification problems. Generally speaking, however, a good loss function can take on much more flexible forms, and should be tailored for…
Entropy is the measure of uncertainty in any data and is adopted for maximisation of mutual information in many remote sensing operations. The availability of wide entropy variations motivated us for an investigation over the suitability…
Machine learning theory has mostly focused on generalization to samples from the same distribution as the training data. Whereas a better understanding of generalization beyond the training distribution where the observed distribution…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
Based on the problem of quantum data compression in a lossless way, we present here an operational interpretation for the family of quantum R\'enyi entropies. In order to do this, we appeal to a very general quantum encoding scheme that…
Although deep learning performs really well in a wide variety of tasks, it still suffers from catastrophic forgetting -- the tendency of neural networks to forget previously learned information upon learning new tasks where previous data is…
The weak law of large numbers implies that, under mild assumptions on the source, the Renyi entropy per produced symbol converges (in probability) towards the Shannon entropy rate. This paper quantifies the speed of this convergence for…
Recently, information theoretic analysis has become a popular framework for understanding the generalization behavior of deep neural networks. It allows a direct analysis for stochastic gradient/Langevin descent (SGD/SGLD) learning…
In this note, we provide analytic expressions for the R\'enyi common information of orders in $(1,\infty)$ for the doubly symmetric binary source (DSBS). Until now, analytic expressions for the R\'enyi common information of all orders in…
In R\'enyi's representation for exponential order statistics, we replace the iid exponential sequence with any iid sequence, and call the resulting order statistic generalized R\'enyi statistic. We prove that by randomly reordering the…
The equivalence between non-extensive C. Tsallis entropy and the extensive entropy introduced by Alfr\'ed R\'enyi is discussed. The R\'enyi entropy is studied from the perspective of the geometry of the Lebesgue and generalised, exotic…
This work provides data-processing and majorization inequalities for $f$-divergences, and it considers some of their applications to coding problems. This work also provides tight bounds on the R\'{e}nyi entropy of a function of a discrete…
Traditionally artificial neural networks (ANNs) are trained by minimizing the cross-entropy between a provided groundtruth delta distribution (encoded as one-hot vector) and the ANN's predictive softmax distribution. It seems, however,…
A range of nonlinear image reconstruction procedures based on extremizing the generalized Shannon entropy, Kullback-Leibler cross-entropy and Renyi information measures and proposed by the author in early papers is presented. The…
It was recently shown that estimating the Shannon entropy $H({\rm p})$ of a discrete $k$-symbol distribution ${\rm p}$ requires $\Theta(k/\log k)$ samples, a number that grows near-linearly in the support size. In many applications $H({\rm…