Related papers: Minimum--Entropy Couplings and their Applications
In this work, we study non-parametric estimation of joint probabilities of a given set of discrete and continuous random variables from their (empirically estimated) 2D marginals, under the assumption that the joint probability could be…
We develop the method of Maximum Entropy (ME) as a technique to generate approximations to probability distributions. The central results consist in (a) justifying the use of relative entropy as the uniquely natural criterion to select a…
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible constrained to match empirical data, for instance, feature expectations. We seek to generalize…
The principle of maximum entropy provides a useful method for inferring statistical mechanics models from observations in correlated systems, and is widely used in a variety of fields where accurate data are available. While the assumptions…
We show how to determine the maximum and minimum possible values of one measure of entropy for a given value of another measure of entropy. These maximum and minimum values are obtained for two standard forms of probability distribution (or…
Given a point $A$ in the convex hull of a given adjoint orbit $\mathcal{O}(F)$ of a compact Lie group $G$, we give a polynomial time algorithm to compute the probability density supported on $\mathcal{O}(F)$ whose expectation is $A$ and…
Two new information-theoretic methods are introduced for establishing Poisson approximation inequalities. First, using only elementary information-theoretic techniques it is shown that, when $S_n=\sum_{i=1}^nX_i$ is the sum of the (possibly…
Let $G_1,\dots, G_m$ be independent Bernoulli random subgraphs of the complete graph ${\cal K}_n$ having variable sizes $x_1,\dots, x_m\in [n]$ and densities $q_1,\dots, q_m\in [0,1]$. Letting $n,m\to+\infty$, we study the connectivity…
Minimum divergence problems under integral constraints appear throughout statistics and probability, including sequential inference, bandit theory, and distributionally robust optimization. In many such settings, dual representations are…
In this paper, the joint distribution of the sum and maximum of independent, not necessarily identically distributed, nonnegative random variables is studied for two cases: i) continuous and ii) discrete random variables. First, a recursive…
The Expectation-Maximization (EM) algorithm for mixture models often results in slow or invalid convergence. The popular convergence proof affirms that the likelihood increases with Q; Q is increasing in the M -step and non-decreasing in…
We consider estimating the Shannon entropy of a discrete distribution $P$ from $n$ i.i.d. samples. Recently, Jiao, Venkat, Han, and Weissman, and Wu and Yang constructed approximation theoretic estimators that achieve the minimax $L_2$…
It is well established that the notion of min-entropy fails to satisfy the \emph{chain rule} of the form $H(X,Y) = H(X|Y)+H(Y)$, known for Shannon Entropy. Such a property would help to analyze how min-entropy is split among smaller blocks.…
Joint probability mass function (PMF) estimation is a fundamental machine learning problem. The number of free parameters scales exponentially with respect to the number of random variables. Hence, most work on nonparametric PMF estimation…
We define and investigate a notion of entropy for quantum error correcting codes. The entropy of a code for a given quantum channel has a number of equivalent realisations, such as through the coefficients associated with the Knill-Laflamme…
Random geometric graphs (RGGs) are commonly used to model networked systems that depend on the underlying spatial embedding. We concern ourselves with the probability distribution of an RGG, which is crucial for studying its random…
Nonuniform subsampling methods are effective to reduce computational burden and maintain estimation efficiency for massive data. Existing methods mostly focus on subsampling with replacement due to its high computational efficiency. If the…
An axiomatic foundation regarding the entropy for complex systems is established. Missing from decades of research was the requirement that entropy must measure the uncertainty at the informational scale of the maximizing distribution,…
Many quantum information measures can be written as an optimization of the quantum relative entropy between sets of states. For example, the relative entropy of entanglement of a state is the minimum relative entropy to the set of separable…
A well-known result across information theory, machine learning, and statistical physics shows that the maximum entropy distribution under a mean constraint has an exponential form called the Gibbs-Boltzmann distribution. This is used for…