Related papers: On the R\'{e}nyi Cross-Entropy
The ultimate purpose of the statistical analysis of ordinal patterns is to characterize the distribution of the features they induce. In particular, knowing the joint distribution of the pair Entropy-Statistical Complexity for a large class…
The Sharma-Mittal entropies generalize the celebrated Shannon, R\'enyi and Tsallis entropies. We report a closed-form formula for the Sharma-Mittal entropies and relative entropies for arbitrary exponential family distributions. We…
The definition of weighted entropy allows for easy calculation of the entropy of the mixture of measures. In this paper we investigate the problem of equivalent definition of the general entropy function in weighted form. We show that under…
The top-k error is a common measure of performance in machine learning and computer vision. In practice, top-k classification is typically performed with deep neural networks trained with the cross-entropy loss. Theoretical results indeed…
Phase-space versions of quantum mechanics -- from Wigner's original distribution to modern discrete-qudit constructions -- represent some states with negative quasi-probabilities. Conventional Shannon and R\'enyi entropies become…
In this note, we studied the asymptotic behaviour of the length of the longest common substring for run-length encoded sequences. When the original sequences are generated by an $\alpha$-mixing process with exponential decay (or…
The adversarial vulnerability of deep neural networks (DNNs) has been actively investigated in the past several years. This paper investigates the scale-variant property of cross-entropy loss, which is the most commonly used loss function…
State-of-the-art neural networks are vulnerable to adversarial examples; they can easily misclassify inputs that are imperceptibly different than their training and test data. In this work, we establish that the use of cross-entropy loss…
Shannon entropy is not the only entropy that is relevant to machine-learning datasets, nor possibly even the most important one. Traditional entropies such as Shannon entropy capture information represented by elements' frequencies but not…
Two families of dependence measures between random variables are introduced. They are based on the R\'enyi divergence of order $\alpha$ and the relative $\alpha$-entropy, respectively, and both dependence measures reduce to Shannon's mutual…
A method of estimating the joint probability mass function of a pair of discrete random variables is described. This estimator is used to construct the conditional Shannon-R\'eyni-Tsallis entropies estimates. From there almost sure rates of…
We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the…
It was recently shown that estimating the Shannon entropy $H({\rm p})$ of a discrete $k$-symbol distribution ${\rm p}$ requires $\Theta(k/\log k)$ samples, a number that grows near-linearly in the support size. In many applications $H({\rm…
We consider two types of entropy, namely, Shannon and R\'{e}nyi entropies of the Poisson distribution, and establish their properties as the functions of intensity parameter. More precisely, we prove that both entropies increase with…
Entropy Estimation is an important problem with many applications in cryptography, statistic,machine learning. Although the estimators optimal with respect to the sample complexity have beenrecently developed, there are still some…
This paper derives bounds for two omnipresent information theoretic measures, the Shannon entropy and its complementary dual, the extropy. Based on a large size data set from a logconcave model, the said bounds are obtained for the entropy…
We introduce a new measure of interdependence among the components of a random vector along the main diagonal of the vector copula, i.e. along the line $u_{1}=\ldots=u_{J}$, for $\left(u_{1},\ldots,u_{J}\right)\in\left[0,1\right]^{J}$. Our…
Algorithmic entropy and Shannon entropy are two conceptually different information measures, as the former is based on size of programs and the later in probability distributions. However, it is known that, for any recursive probability…
Although the recent progress is substantial, deep learning methods can be vulnerable to the maliciously generated adversarial examples. In this paper, we present a novel training procedure and a thresholding test strategy, towards robust…
Many axiomatic definitions of entropy, such as the R\'enyi entropy, of a random variable are closely related to the $\ell_{\alpha}$-norm of its probability distribution. This study considers probability distributions on finite sets, and…