Related papers: A series of maximum entropy upper bounds of the di…
This paper derives new bounds on the difference of the entropies of two discrete random variables in terms of the local and total variation distances between their probability mass functions. The derivation of the bounds relies on maximal…
We describe a method to computationally estimate the probability density function of a univariate random variable by applying the maximum entropy principle with some local conditions given by Gaussian functions. The estimation errors and…
A novel, non-trivial, probabilistic upper bound on the entropy of an unknown one-dimensional distribution, given the support of the distribution and a sample from that distribution, is presented. No knowledge beyond the support of the…
We calculate the average differential entropy of a $q$-component Gaussian mixture in $\mathbb R^n$. For simplicity, all components have covariance matrix $\sigma^2 {\mathbf 1}$, while the means $\{\mathbf{W}_i\}_{i=1}^{q}$ are i.i.d.…
In this paper, we derive generic bounds on the maximum deviations in prediction errors for sequential prediction via an information-theoretic approach. The fundamental bounds are shown to depend only on the conditional entropy of the data…
The total variation distance is a core statistical distance between probability measures that satisfies the metric axioms, with value always falling in $[0,1]$. This distance plays a fundamental role in machine learning and signal…
The problem of determining the joint probability distributions for correlated random variables with pre-specified marginals is considered. When the joint distribution satisfying all the required conditions is not unique, the "most unbiased"…
Mixture distributions are a workhorse model for multimodal data in information theory, signal processing, and machine learning. Yet even when each component density is simple, the differential entropy of the mixture is notoriously hard to…
Gaussian mixture models are widely used in Statistics. A fundamental aspect of these distributions is the study of the local maxima of the density, or modes. In particular, it is not known how many modes a mixture of $k$ Gaussians in $d$…
This work contains two single-letter upper bounds on the entropy rate of a discrete-valued stationary stochastic process, which only depend on second-order statistics, and are primarily suitable for models which consist of relatively large…
In this paper relations among some kinds of cumulative entropies and moments of order statistics are presented. By using some characterizations and the symmetry of a non negative and absolutely continuous random variable X, lower and upper…
Gaussian mixture distributions are commonly employed to represent general probability distributions. Despite the importance of using Gaussian mixtures for uncertainty estimation, the entropy of a Gaussian mixture cannot be calculated…
The conditional mean is a fundamental and important quantity whose applications include the theories of estimation and rate-distortion. It is also notoriously difficult to work with. This paper establishes novel bounds on the differential…
A well-known result across information theory, machine learning, and statistical physics shows that the maximum entropy distribution under a mean constraint has an exponential form called the Gibbs-Boltzmann distribution. This is used for…
Maximum entropy estimation is of broad interest for inferring properties of systems across many different disciplines. In this work, we significantly extend a technique we previously introduced for estimating the maximum entropy of a set of…
Motivated by the entropy computations relevant to the evaluation of decrease in entropy in bit reset operations, the authors investigate the deficit in an entropic inequality involving two independent random variables, one continuous and…
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably…
Many inference problems involving questions of optimality ask for the maximum or the minimum of a finite set of unknown quantities. This technical report derives the first two posterior moments of the maximum of two correlated Gaussian…
We consider variational problem related to entropy maximization in the two-dimensional Euler equations, in order to investigate the long-time dynamics of solutions with bounded vorticity. Using variations on the classical min-max principle…
In this article, we are proposing a closed-form solution for the capacity of the single quantum channel. The Gaussian distributed input has been considered for the analytical calculation of the capacity. In our previous couple of papers, we…