Related papers: Practical Estimation of Renyi Entropy
Efficient approximation lies at the heart of large-scale machine learning problems. In this paper, we propose a novel, robust maximum entropy algorithm, which is capable of dealing with hundreds of moments and allows for computationally…
Relative entropy is a measure of distinguishability for quantum states, and plays a central role in quantum information theory. The family of Renyi entropies generalizes to Renyi relative entropies that include as special cases most entropy…
Inferring models, predicting the future, and estimating the entropy rate of discrete-time, discrete-event processes is well-worn ground. However, a much broader class of discrete-event processes operates in continuous-time. Here, we provide…
This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in…
We present an algorithm to estimate the configurational entropy $S$ of a polymer. The algorithm uses the statistics of coincidences among random samples of configurations and is related to the catch-tag-release method for estimation of…
Analytic continuation of numerical data obtained in imaginary time or frequency has become an essential part of many branches of quantum computational physics. It is, however, an ill-conditioned procedure and thus a hard numerical problem.…
Entropy rate is a real valued functional on the space of discrete random sources which lacks a closed formula even for subclasses of sources which have intuitive parameterizations. A good way to overcome this problem is to examine its…
Entropy is a famous and well established concept in physics and engineering that can be used for explanation of basic fundamentals as well it finds applications in several areas, from quantum physics to astronomy, from network communication…
One of the main problems in cryptography is to give criteria to provide good comparators of cipher systems. The security of a cipher system must include the security of the algorithm, the security of the key generator and management module…
The translation of written language has been known since the 3rd century BC; however, its necessity has become increasingly common in the information age. Today, many translators exist, based on encoder-decoder deep architectures,…
This paper considers an information bottleneck problem with the objective of obtaining a most informative representation of a hidden feature subject to a R\'enyi entropy complexity constraint. The optimal bottleneck trade-off between…
Compression of documents, images, audios and videos have been traditionally practiced to increase the efficiency of data storage and transfer. However, in order to process or carry out any analytical computations, decompression has become…
We consider the problem of finite sample corrections for entropy estimation. New estimates of the Shannon entropy are proposed and their systematic error (the bias) is computed analytically. We find that our results cover correction…
This work includes a number of novel contributions for the multiple-source adaptation problem. We present new normalized solutions with strong theoretical guarantees for the cross-entropy loss and other similar losses. We also provide new…
Entropy and its various generalizations are important in many fields, including mathematical statistics, communication theory, physics and computer science, for characterizing the amount of information associated with a probability…
Shannon and Renyi entropies are quantitative measures of uncertainty in a data set. They are developed by Renyi in the context of entropy theory. These measures have been studied in the case of the multivariate t-distributions. We extend…
We consider the task of estimating the entropy of $k$-ary distributions from samples in the streaming model, where space is limited. Our main contribution is an algorithm that requires $O\left(\frac{k \log…
We address the problem of estimating the running quantile of a data stream when the memory for storing observations is limited. We (i) highlight the limitations of approaches previously described in the literature which make them unsuitable…
To quantify the complexity of a system, entropy-based methods have received considerable critical attentions in real-world data analysis. Among numerous entropy algorithms, amplitude-based formulas, represented by Sample Entropy, suffer…
Entropy regularization is commonly used to improve policy optimization in reinforcement learning. It is believed to help with \emph{exploration} by encouraging the selection of more stochastic policies. In this work, we analyze this claim…