Related papers: Dimension-Free Empirical Entropy Estimation
We present novel bounds for estimating discrete probability distributions under the $\ell_\infty$ norm. These are nearly optimal in various precise senses, including a kind of instance-optimality. Our data-dependent convergence guarantees…
This paper introduces a numerical framework for establishing lower bounds on the conditional von-Neumann entropy in device-independent quantum cryptography and randomness extraction scenarios. Leveraging a hierarchy of semidefinite programs…
We present a detailed derivation of some estimators of Shannon entropy for discrete distributions. They hold for finite samples of N points distributed into M "boxes", with N and M -> oo, but N/M < oo. In the high sampling regime (<< 1…
In the paper, we introduce the maximum entropy estimator based on 2-dimensional empirical distribution of the observation sequence of hidden Markov model , when the sample size is big: in that case computing the maximum likelihood estimator…
The problem of Shannon entropy estimation in countable infinite alphabets is addressed from the study and use of convergence results of the entropy functional, which is known to be discontinuous with respect to the total variation distance…
We present some new nonparametric estimators of entropies and we establish almost sure consistency and central limit Theorems for some of the most important entropies in the discrete case. Our theorical results are validated by simulations.
Information theory on a time-discrete setting in the framework of time series analysis is generalized to the time-continuous case. Considerations of the Roessler and Lorenz dynamics as well as the Ornstein-Uhlenbeck process yield for…
A broad set of sufficient conditions that guarantees the existence of the maximum entropy (maxent) distribution consistent with specified bounds on certain generalized moments is derived. Most results in the literature are either focused on…
Dependence among marginally constrained observations can break a finite-sample barrier. To formalize this phenomenon, we introduce the \emph{minimum list entropy coupling} $H(P\|Q_1,\dots,Q_m)$, the minimum conditional entropy…
In this paper, we develop an approach for the exact determination of the minimum sample size for estimating the parameter of an integer-valued random variable, which is parameterized by its expectation. Under some continuity and unimodal…
Diffusion models perform remarkably well on high-dimensional data such as images, often using only a modest number of reverse-time steps. Despite this practical success, existing convergence theory does not fully explain why such samplers…
We consider the problem of guessing the realization of a random variable but under more general Tsallis' non-extensive entropic framework rather than the classical Maxwell-Boltzman-Gibbs-Shannon framework. We consider both the conditional…
A common statistical situation concerns inferring an unknown distribution Q(x) from a known distribution P(y), where X (dimension n), and Y (dimension m) have a known functional relationship. Most commonly, n<m, and the task is relatively…
We consider distributions of ordered random vectors with given one-dimensional marginal distributions. We give an elementary necessary and sufficient condition for the existence of such a distribution with finite entropy. In this case, we…
The ability of many powerful machine learning algorithms to deal with large data sets without compromise is often hampered by computationally expensive linear algebra tasks, of which calculating the log determinant is a canonical example.…
We examine the minimization of information entropy for measures on the phase space of bounded domains, subject to constraints that are averages of grand canonical distributions. We describe the set of all such constraints and show that it…
We prove a variety of new and refined uniform continuity bounds for entropies of both classical random variables on an infinite state space and of quantum states of infinite-dimensional systems. We obtain the first tight continuity estimate…
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible while constrained to match empirically estimated feature expectations. However, in many real-world…
Here we present an analytic approximation for the entropy of floating-point numbers, along with bounds on the error of this approximation. It is well-known that the differential entropy is tightly linked to the discrete entropy of a…
Entropy is useful in statistical problems as a measure of irreversibility, randomness, mixing, dispersion, and number of microstates. However, there remains ambiguity over the precise mathematical formulation of entropy, generalized beyond…