Related papers: Dimension-Free Empirical Entropy Estimation

The classical problem of maximizing the Shannon entropy of a sum of independent random variables supported on a finite alphabet is considered and settled in the ternary case. Namely, the following theorem is established: if…

Maximum entropy distributions with discrete support in $m$ dimensions arise in machine learning, statistics, information theory, and theoretical computer science. While structural and computational properties of max-entropy distributions…

This paper addresses the challenge of identifying a minimal subset of discrete, independent variables that best predicts a binary class. We propose an efficient iterative method that sequentially selects variables based on which one…

We present a new class of estimators of Shannon entropy for severely undersampled discrete distributions. It is based on a generalization of an estimator proposed by T. Schuermann, which itself is a generalization of an estimator proposed…

This paper addresses a fundamental problem in random variate generation: given access to a random source that emits a stream of independent fair bits, what is the most accurate and entropy-efficient algorithm for sampling from a discrete…

When data contains measurement errors, it is necessary to make assumptions relating the observed, erroneous data to the unobserved true phenomena of interest. These assumptions should be justifiable on substantive grounds, but are often…

Maximum entropy method is a constructive criterion for setting up a probability distribution maximally non-committal to missing information on the basis of partial knowledge, usually stated as constrains on expectation values of some…

Maximization of an expensive, unimodal function under random observations has been an important problem in hyperparameter tuning. It features expensive function evaluations (which means small budgets) and a high level of noise. We develop…

The aim of this paper is to provide several novel upper bounds on the excess risk with a primal focus on classification problems. We suggest two approaches and the obtained bounds are represented via the distribution dependent local…

Maximum entropy models provide the least constrained probability distributions that reproduce statistical properties of experimental datasets. In this work we characterize the learning dynamics that maximizes the log-likelihood in the case…

Entropy, its production, and its change in a dynamical system can be understood from either a fully stochastic dynamic description or from a deterministic dynamics exhibiting chaotic behavior. By taking the former approach based on the…

I show that the maximum entropy principle can be replaced by a more natural assumption, that there exists a phenomenological function of entropy consistent with the microscopic model. The requirement of existence provides then a unique…

In classical density (or density-functional) estimation, it is standard to assume that the underlying distribution has a density with respect to the Lebesgue measure. However, when the data distribution is a mixture of continuous and…

Given a knowledge base KB containing first-order and statistical facts, we consider a principled method, called the random-worlds method, for computing a degree of belief that some formula Phi holds given KB. If we are reasoning about a…

We consider shift spaces in which elements of the alphabet may overlap nontransitively. We define a notion of entropy for such spaces, give several techniques for computing lower bounds for it, and show that it is equal to a limit of…

We introduce an independence criterion based on entropy regularized optimal transport. Our criterion can be used to test for independence between two samples. We establish non-asymptotic bounds for our test statistic and study its…

In this paper we derive the maximum entropy characteristics of a particular rank order distribution, namely the discrete generalized beta distribution, which has recently been observed to be extremely useful in modelling many several…

Language prediction is constrained by informational entropy intrinsic to language, such that there exists a limit to how accurate any language model can become and equivalently a lower bound to language compression. The most efficient…

Entropy rate is a real valued functional on the space of discrete random sources which lacks a closed formula even for subclasses of sources which have intuitive parameterizations. A good way to overcome this problem is to examine its…

Inferring models, predicting the future, and estimating the entropy rate of discrete-time, discrete-event processes is well-worn ground. However, a much broader class of discrete-event processes operates in continuous-time. Here, we provide…