Related papers: Practical Estimation of Renyi Entropy
We study the problem of maximizing R{\'e}nyi entropy of order $2$ (equivalently, minimizing the index of coincidence) over the set of joint distributions with prescribed marginals. A closed-form optimizer is known under a feasibility…
We propose a new method to combine adaptive processes with a class of entropy estimators for the case of streams of data. Starting from a first estimation obtained from a batch of initial data, model parameters are estimated at each step by…
In this paper, we study streaming and online algorithms in the context of randomness in the input. For several problems, a random order of the input sequence---as opposed to the worst-case order---appears to be a necessary evil in order to…
In this paper, I expand Shannon's definition of entropy into a new form of entropy that allows integration of information from different random events. Shannon's notion of entropy is a special case of my more general definition of entropy.…
The Random Permutation Set (RPS) is a new type of set proposed recently, which can be regarded as the generalization of evidence theory. To measure the uncertainty of RPS, the entropy of RPS and its corresponding maximum entropy have been…
This paper presents a command-line tool, called Entropia, that implements a family of conformance checking measures for process mining founded on the notion of entropy from information theory. The measures allow quantifying classical…
Token-level reweighting is a simple yet effective mechanism for controlling supervised fine-tuning, but common indicators are largely one-dimensional: the ground-truth probability reflects downstream alignment, while token entropy reflects…
The last decade has seen a number of advances in computationally efficient algorithms for statistical methods subject to robustness constraints. An estimator may be robust in a number of different ways: to contamination of the dataset, to…
The Principle of Maximum Entropy is a rigorous technique for estimating an unknown distribution given partial information while simultaneously minimizing bias. However, an important requirement for applying the principle is that the…
We explore a method of statistical estimation called Maximum Entropy on the Mean (MEM) which is based on an information-driven criterion that quantifies the compliance of a given point with a reference prior probability measure. At the core…
Feature selection, in the context of machine learning, is the process of separating the highly predictive feature from those that might be irrelevant or redundant. Information theory has been recognized as a useful concept for this task, as…
The classical problem of moments is addressed by the maximum entropy approach for one-dimensional discrete distributions. The numerical technique of adaptive support approximation is proposed to reconstruct the distributions in the region…
The classical combinatorics-based password strength formula provides a result in tens of bits, whereas the NIST Entropy Estimation Suite give a result between 0 and 1 for Min-entropy. In this work, we present a newly developed metric --…
Moment-closure methods are popular tools to simplify the mathematical analysis of stochastic models defined on networks, in which high dimensional joint distributions are approximated (often by some heuristic argument) as functions of lower…
We describe a quantum algorithm to estimate the $\alpha$-Renyi entropy of an unknown density matrix $\rho\in\mathcal{C}^{d\times d}$ for $\alpha\neq 1$ by combining the recent technique of quantum singular value transformations with the…
We discuss the properties of invariant measures corresponding to iterated function systems (IFSs) with place-dependent probabilities and compute their Renyi entropies, generalized dimensions, and multifractal spectra. It is shown that with…
A definition of entropy via the Kolmogorov algorithmic complexity is discussed. As examples, we show how the meanfield theory for the Ising model, and the entropy of a perfect gas can be recovered. The connection with computations are…
In vector quantization the number of vectors used to construct the codebook is always an undefined problem, there is always a compromise between the number of vectors and the quantity of information lost during the compression. In this text…
We develop a maximum relative entropy formalism to generate optimal approximations to probability distributions. The central results consist in (a) justifying the use of relative entropy as the uniquely natural criterion to select a…
For a closed-loop control system with a digital channel between the sensor and the controller, the notion of invariance entropy quantifies the smallest average rate of information transmission above which a given compact subset of the state…