Related papers: Extraction of physical laws from joint experimenta…
We study the problem of discovering the simplest latent variable that can make two observed discrete variables conditionally independent. The minimum entropy required for such a latent is known as common entropy in information theory. We…
We examine a class of deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical…
Let $X$ be a non-negative random variable and let the conditional distribution of a random variable $Y$, given $X$, be ${Poisson}(\gamma \cdot X)$, for a parameter $\gamma \geq 0$. We identify a natural loss function such that: 1) The…
A composite likelihood is an inference function derived by multiplying a set of likelihood components. This approach provides a flexible framework for drawing inference when the likelihood function of a statistical model is computationally…
The analysis of Tables of particle properties shows that the probability distribution of the results of physical measurements is far from the conventional Gaussian $\rho(\xi)=exp(-\xi^2/2) $, but is more likely to follow the simple…
A classical problem of statistical inference is the valid specification of a model that can account for the statistical dependencies between observations when the true structure is dense, intractable, or unknown. To address this problem, a…
In this paper, the joint distribution of the sum and maximum of independent, not necessarily identically distributed, nonnegative random variables is studied for two cases: i) continuous and ii) discrete random variables. First, a recursive…
This paper derives new bounds on the difference of the entropies of two discrete random variables in terms of the local and total variation distances between their probability mass functions. The derivation of the bounds relies on maximal…
The extropy is a measure of information introduced by Lad et al. (2015) as dual to entropy. As the entropy, it is a shift-independent information measure. We introduce here the notion of weighted extropy, a shift-dependent information…
Mutual information is widely used, in a descriptive way, to measure the stochastic dependence of categorical random variables. In order to address questions such as the reliability of the descriptive value, one must consider…
We discuss the connection between information and copula theories by showing that a copula can be employed to decompose the information content of a multivariate distribution into marginal and dependence components, with the latter…
This paper develops a new divergence that generalizes relative entropy and can be used to compare probability measures without a requirement of absolute continuity. We establish properties of the divergence, and in particular derive and…
Mixture distributions are a workhorse model for multimodal data in information theory, signal processing, and machine learning. Yet even when each component density is simple, the differential entropy of the mixture is notoriously hard to…
A generic algorithm for the extraction of probabilistic (Bayesian) information about model parameters from data is presented. The algorithm propagates an ensemble of particles in the product space of model parameters and outputs. Each…
In probability density function (PDF) methods a transport equation is solved numerically to compute the time and space dependent probability distribution of several flow variables in a turbulent flow. The joint PDF of the velocity…
The additional information carried by an enlarged filtration and its measurement was studied by several authors. Already Meyer (Sur un theoreme de J. Jacod, 1978) and Yor (Entropie d'une partition, et grossissement initial d'une filtration,…
A new concept of the available force is proposed to investigate the performance of the complex systems having long-range interactions. Since the covariance of average velocity in double time interval and available force equals zero, it is…
Quantum superposition, a cornerstone of quantum mechanics, enables systems to exist in multiple states simultaneously, giving rise to probabilistic outcomes. In quantum information science, conditional entropy has become a key metric for…
Mutual information is widely used in artificial intelligence, in a descriptive way, to measure the stochastic dependence of discrete random variables. In order to address questions such as the reliability of the empirical value, one must…
Mutual information is widely used in artificial intelligence, in a descriptive way, to measure the stochastic dependence of discrete random variables. In order to address questions such as the reliability of the empirical value, one must…