Related papers: Infinite Divisibility of Information
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of…
We consider the problem of decomposing the total mutual information conveyed by a pair of predictor random variables about a target random variable into redundant, unique and synergistic contributions. We focus on the relationship between…
Information collection is a fundamental problem in big data, where the size of sampling sets plays a very important role. This work considers the information collection process by taking message importance into account. Similar to…
The information in an individual finite object (like a binary string) is commonly measured by its Kolmogorov complexity. One can divide that information into two parts: the information accounting for the useful regularity present in the…
This paper examines the joint problem of detection and identification of a sudden and unobservable change in the probability distribution function (pdf) of a sequence of independent and identically distributed (i.i.d.) random variables to…
Recent developments in extracting and processing biological and clinical data are allowing quantitative approaches to studying living systems. High-throughput sequencing, expression profiles, proteomics, and electronic health records are…
In a previous report we have evaluated analytically the mutual information between the firing rates of N independent units and a set of multi-dimensional continuous+discrete stimuli, for a finite population size and in the limit of large…
A collaborative distributed binary decision problem is considered. Two statisticians are required to declare the correct probability measure of two jointly distributed memoryless process, denoted by $X^n=(X_1,\dots,X_n)$ and…
Symmetry of information establishes a relation between the information that x has about y (denoted I(x : y)) and the information that y has about x (denoted I(y : x)). In classical information theory, the two are exactly equal, but in…
We provide finite-sample distribution approximations, that are uniform in the parameter, for inference in linear mixed models. Focus is on variances and covariances of random effects in cases where existing theory fails because their…
Each compact manifold M of finite dimension k is differentiable and supports an intrinsic probability measure. There then exists a measurable transformation of M to the k-dimensional "surface" of the (k+1)-dimensional ball.
Let $\mathsf{N}_{\rm d}\left[X\right]=\frac{1}{2\pi {\rm e}}{\rm e}^{2\mathsf{H}\left[X\right]}$ denote the entropy power of the discrete random variable $X$ where $\mathsf{H}\left[X\right]$ denotes the discrete entropy of $X$. In this…
After reviewing a large body of literature on the modeling of bivariate discrete distributions with finite support, \cite{Gee20} made a compelling case for the use of $I$-projections in the sense of \cite{Csi75} as a sound way to attempt to…
Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a…
Inferring the causal structure of a set of random variables from a finite sample of the joint distribution is an important problem in science. Recently, methods using additive noise models have been suggested to approach the case of…
Let ${\cal I}$ be the set of all infinitely divisible random variables\ with finite second moments, ${\cal I}_0=\{X\in{\cal I}:{\rm Var}(X)>0\}$, $P_{\cal I}=\inf_{X\in{\cal I}}P\{|X-E[X]|\le \sqrt{{\rm Var}(X)}\}$ and $P_{{\cal…
We investigate the concept of entropy in probabilistic theories more general than quantum mechanics, with particular reference to the notion of information causality recently proposed by Pawlowski et. al. (arXiv:0905.2992). We consider two…
Given i.i.d. data from an unknown distribution, we consider the problem of predicting future items. An adaptive way to estimate the probability density is to recursively subdivide the domain to an appropriate data-dependent granularity. A…
We consider finite blocklength lossy compression of information sources whose components are independent but non-identically distributed. Crucially, Gaussian sources with memory and quadratic distortion can be cast in this form. We show…
We study conditional mutual information (cMI) between a pair of variables $X,Y$ given a third one $Z$ and derived quantities including transfer entropy (TE) and causation entropy (CE) in the dynamically relevant context where $X=T(Y,Z)$ is…