Related papers: Stochastic comparisons, differential entropy and v…
This paper demonstrates that basic statistics (mean, variance) of the logarithm of the variate itself can be used in the calculation of differential entropy among random variables known to be multiples and powers of a common underlying…
We propose an additional category of dimensionless groups based on the principle of {\it entropic similarity}, defined by ratios of (i) entropy production terms; (ii) entropy flow rates or fluxes; or (iii) information flow rates or fluxes.…
The article derives some novel independence measures and contrast functions for Blind Source Separation (BSS) application. For the $k^{th}$ order differentiable multivariate functions with equal hyper-volumes (region bounded by…
The entropy of a pair of random variables is commonly depicted using a Venn diagram. This representation is potentially misleading, however, since the multivariate mutual information can be negative. This paper presents new measures of…
We study the one-point probability distribution functions (PDFs) of the peculiar velocity and the density fluctuation in a cosmological fluid. Within the perturbative approach to the structure formation scenario, the effect of ``pressure''…
These lectures deal with the problem of inductive inference, that is, the problem of reasoning under conditions of incomplete information. Is there a general method for handling uncertainty? Or, at least, are there rules that could in…
In its continuous version, the entropy functional measuring the information content of a given probability density may be plagued by a "measure" problem that results from improper weighting of phase space. This issue is addressed…
Permutation entropy quantifies the diversity of possible orderings of the values a random or deterministic system can take, as Shannon entropy quantifies the diversity of values. We show that the metric and permutation entropy…
Estimating density ratios between pairs of intractable data distributions is a core problem in probabilistic modeling, enabling principled comparisons of sample likelihoods under different data-generating processes across conditions and…
Computational pseudorandomness studies the extent to which a random variable $\bf{Z}$ looks like the uniform distribution according to a class of tests $\cal{F}$. Computational entropy generalizes computational pseudorandomness by studying…
If the prior probability distributions of all possible hypothetical true means and all possible observed means of a continuous variable are conditional on the universal set of all numbers (i.e., before the nature of a study is known and a…
Rooted trees with probabilities are used to analyze properties of a variable length code. A bound is derived on the difference between the entropy rates of the code and a memoryless source. The bound is in terms of normalized informational…
The entropy power inequality for independent random vectors is a foundational result of information theory, with deep connections to probability and geometric functional analysis. Several extensions of the entropy power inequality have been…
We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models,…
We consider the problem of learning a target probability distribution over a set of $N$ binary variables from the knowledge of the expectation values (with this target distribution) of $M$ observables, drawn uniformly at random. The space…
We give a variational formulation of classical statistical mechanics where the one-body density and the local entropy distribution constitute the trial fields. Using Levy's constrained search method it is shown that the grand potential is a…
We conduct numerical experiments to determine the density probability distribution function (PDF) produced in supersonic, isothermal, self-gravitating turbulence of the sort that is ubiquitous in star-forming molecular clouds. Our…
General characterization of physical measurements is discussed within the framework of a classical information theory. Uncertainty relation for simultaneous measurements of two physical observables is defined in this framework for…
We consider a "length-biased" shift-dependent information measure, related to the differential entropy in which higher weight is assigned to large values of observed random variables. This allows us to introduce the notions of "weighted…
Understanding geometric properties of natural language processing models' latent spaces allows the manipulation of these properties for improved performance on downstream tasks. One such property is the amount of data spread in a model's…