Related papers: R\'{e}nyi Cross-Entropy Measures for Common Distri…
R\'enyi divergence is related to R\'enyi entropy much like information divergence (also called Kullback-Leibler divergence or relative entropy) is related to Shannon's entropy, and comes up in many settings. It was introduced by R\'enyi as…
We propose a permutation-invariant loss function designed for the neural networks reconstructing a set of elements without considering the order within its vector representation. Unlike popular approaches for encoding and decoding a set,…
The cross-entropy method is a simple but efficient method for global optimization. In this paper we provide two online variants of the basic CEM, together with a proof of convergence.
Escort distributions have been shown to be very useful in a great variety of fields ranging from information theory, nonextensive statistical mechanics till coding theory, chaos and multifractals. In this work we give the notion and the…
Entropic uncertainty relations are interesting in their own rights as well as for a lot of applications. Keeping this in mind, we try to make the corresponding inequalities as tight as possible. The use of parametrized entropies also allows…
Recently, two extensions of Wyner's common information\textemdash exact and R\'enyi common informations\textemdash were introduced respectively by Kumar, Li, and El Gamal (KLE), and the present authors. The class of common information…
In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of uniform feature and class distributions. It belongs to…
Tsallis and R\'{e}nyi entropy measures are two possible different generalizations of the Boltzmann-Gibbs entropy (or Shannon's information) but are not generalizations of each others. It is however the Sharma-Mittal measure, which was…
Two maximization problems of R\'enyi entropy rate are investigated: the maximization over all stochastic processes whose marginals satisfy a linear constraint, and the Burg-like maximization over all stochastic processes whose…
Entropy is a measure of self-information which is used to quantify losses. Entropy was developed in thermodynamics, but is also used to compare probabilities based on their deviating information content. Corresponding model uncertainty is…
We study the continuity property of the generalized entropy as a function of the underlying probability distribution, defined with an action space and a loss function, and use this property to answer the basic questions in statistical…
Exponential families are widely used in machine learning, including many distributions in continuous and discrete domains (e.g., Gaussian, Dirichlet, Poisson, and categorical distributions via the softmax transformation). Distributions in…
Entanglement entropy is an essential metric for characterizing quantum many-body systems, but its numerical evaluation for neural network representations of quantum states has so far been inefficient and demonstrated only for the restricted…
We show that in the continuum limit, the average spectrum method (ASM) is equivalent to maximizing R\'enyi entropies of order $\eta$, of which Shannon entropy is the special case $\eta=1$. The order of R\'enyi entropy is determined by the…
This article presents a new class of generalized transmuted lifetime distributions which includes a large number of lifetime distributions as sub-family. Several important mathematical quantities such as density function, distribution…
The concept of Entropy plays a key role in Information Theory, Statistics, and Machine Learning.This paper introduces a new entropy measure, called the t-entropy, which exploits the concavity of the inverse-tan function. We analytically…
Relations between Shannon entropy and Renyi entropies of integer order are discussed. For any N-point discrete probability distribution for which the Renyi entropies of order two and three are known, we provide an lower and an upper bound…
We describe a method to estimate R\'enyi entanglement entropy of a spin system, which is based on the replica trick and generative neural networks with explicit probability estimation. It can be extended to any spin system or lattice field…
In this study, the quantum R\'{e}nyi entropy power inequality of order $p>1$ and power $\kappa$ is introduced as a quantum analog of the classical R\'{e}nyi-$p$ entropy power inequality. To derive this inequality, we first exploit the…
Entropy Estimation is an important problem with many applications in cryptography, statistic,machine learning. Although the estimators optimal with respect to the sample complexity have beenrecently developed, there are still some…