Related papers: An Equal-Probability Partition of the Sample Space…
The Shannon entropy, and related quantities such as mutual information, can be used to quantify uncertainty and relevance. However, in practice, it can be difficult to compute these quantities for arbitrary probability distributions,…
We present estimators for entropy and other functions of a discrete probability distribution when the data is a finite sample drawn from that probability distribution. In particular, for the case when the probability distribution is a joint…
We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the…
We study random points on the real line generated by the eigenvalues in unitary invariant random matrix ensembles or by more general repulsive particle systems. As the number of points tends to infinity, we prove convergence of the…
We study two types of probability measures on the set of integer partitions of $n$ with at most $m$ parts. The first one chooses the random partition with a chance related to its largest part only. We then obtain the limiting distributions…
Reliable data-driven estimation of Shannon entropy from small data sets, where the number of examples is potentially smaller than the number of possible outcomes, is a critical matter in several applications. In this paper, we introduce a…
We present a detailed derivation of some estimators of Shannon entropy for discrete distributions. They hold for finite samples of N points distributed into M "boxes", with N and M -> oo, but N/M < oo. In the high sampling regime (<< 1…
The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space,…
The empirical distribution function assigns mass $1/n$ to each of the $n$ observations in a sample. As these are highly variable, estimation error may be reduced by replacing them with estimated observations that are asymptotically less…
For studies in reliability, biometry, and survival analysis, the length-biased distribution is often well-suited for certain natural sampling plans. In this paper, we study the strong uniform consistency of two nonparametric estimators for…
We propose a nonparametric estimator of multivariate joint entropy based on partitioned sample spacing (PSS). The method extends univariate spacing ideas to $\mathbb{R}^{d}$ by partitioning into localized cells and aggregating within-cell…
We consider the problem of estimating the missing mass, partition function or evidence and its probability distribution in the case that for each sample point in the discrete sample space its (unnormalized) probability mass is revealed.…
In this paper we will analyze discrete probability distributions in which probabilities of particular outcomes of some experiment (microstates) can be represented by the ratio of natural numbers (in other words, probabilities are…
The logical basis for information theory is the newly developed logic of partitions that is dual to the usual Boolean logic of subsets. The key concept is a "distinction" of a partition, an ordered pair of elements in distinct blocks of the…
An integer partition of $n$ is a decreasing sequence of positive integers that add up to $[n]$. Back in $1979$ Macdonald posed a question about the limit value of the probability that two partitions chosen uniformly at random, and…
Two closely related discrete probability distributions are introduced. In each case the support is a set of vectors in $\mathbb{R}^n$ obtained from the partitions of the fixed positive integer $n$. These distributions arise naturally when…
In Bayesian inference, we seek to compute information about random variables such as moments or quantiles on the basis of {available data} and prior information. When the distribution of random variables is {intractable}, Monte Carlo (MC)…
In this paper, we introduce a flexible and widely applicable nonparametric entropy-based testing procedure that can be used to assess the validity of simple hypotheses about a specific parametric population distribution. The testing…
We present novel bounds for estimating discrete probability distributions under the $\ell_\infty$ norm. These are nearly optimal in various precise senses, including a kind of instance-optimality. Our data-dependent convergence guarantees…
Even if a probability distribution is properly normalizable, its associated Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze conditions under which this phenomenon can occur. Roughly speaking, this happens when…