Related papers: Information Bottleneck on General Alphabets

When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the…

We propose two new measures for extracting the unique information in $X$ and not $Y$ about a message $M$, when $X, Y$ and $M$ are joint random variables with a given joint distribution. We take a Markov based approach, motivated by…

Numerous deep learning algorithms have been inspired by and understood via the notion of information bottleneck, where unnecessary information is (often implicitly) minimized while task-relevant information is maximized. However, a rigorous…

This paper shows that for any random variables $X$ and $Y$, it is possible to represent $Y$ as a function of $(X,Z)$ such that $Z$ is independent of $X$ and $I(X;Z|Y)\le\log(I(X;Y)+1)+4$ bits. We use this strong functional representation…

Consider a sequence of Poisson random connection models (X_n,lambda_n,g_n) on R^d, where lambda_n / n^d \to lambda > 0 and g_n(x) = g(nx) for some non-increasing, integrable connection function g. Let I_n(g) be the number of isolated…

Previous work has shown that DNNs with large depth $L$ and $L_{2}$-regularization are biased towards learning low-dimensional representations of the inputs, which can be interpreted as minimizing a notion of rank $R^{(0)}(f)$ of the learned…

Let $f:\mathbb{R}^k\to \mathbb{R}$ be a measurable function, and let $\{U_i\}_{i\in\mathbb{N}}$ be a sequence of i.i.d. random variables. Consider the random process $Z_i=f(U_{i},...,U_{i+k-1})$. We show that for all $\ell$, there is a…

In this work, lossy distributed compression of pairs of correlated sources is considered. Conventionally, Shannon's random coding arguments -- using randomly generated unstructured codebooks whose blocklength is taken to be asymptotically…

The Glivenko--Cantelli theorem is a uniform version of the strong law of large numbers. It states that for every IID sequence of random variables, the empirical measure converges to the underlying distribution (in the sense of uniform…

\cite{HillMotegi2017} present a new general asymptotic theory for the maximum of a random array $\{\mathcal{X}_{n}(i)$ $:$ $1$ $\leq $ $i$ $\leq $ $\mathcal{L}\}_{n\geq 1}$, where each $\mathcal{X}_{n}(i)$ is assumed to converge in…

This paper deals with sequences of random variables $X_n$ only taking values in $\{0,\ldots,n\}$. The probability generating functions of such random variables are polynomials of degree $n$. Under the assumption that the roots of these…

Let $f$ be a transcendental entire function and let $I(f)$ denote the set of points that escape to infinity under iteration. We give conditions which ensure that, for certain functions, $I(f)$ is connected. In particular, we show that…

Consider informative selection of a sample from a finite population. Responses are realized as independent and identically distributed (i.i.d.) random variables with a probability density function (p.d.f.) f, referred to as the…

This study comes as a timely response to mounting criticism of the information bottleneck (IB) theory, injecting fresh perspectives to rectify misconceptions and reaffirm its validity. Firstly, we introduce an auxiliary function to…

Information Theory (IT) has been used in Machine Learning (ML) from early days of this field. In the last decade, advances in Deep Neural Networks (DNNs) have led to surprising improvements in many applications of ML. The result has been a…

In this paper, we consider U-statistics whose data is a strictly stationary sequence which can be expressed as a functional of an i.i.d. one. We establish a strong law of large numbers, a bounded law of the iterated logarithms and a central…

We consider general discrete Markov Random Fields(MRFs) with additional bottleneck potentials which penalize the maximum (instead of the sum) over local potential value taken by the MRF-assignment. Bottleneck potentials or analogous…

We prove that a suitably de-biased version of Chatterjee's rank correlation based on i.i.d. copies of a random vector $(X,Y)$ is asymptotically normal whenever $Y$ is not almost surely constant. No further conditions on the joint…

The likelihood encoder with a random codebook is demonstrated as an effective tool for source coding. Coupled with a soft covering lemma (associated with channel resolvability), likelihood encoders yield simple achievability proofs for…

Given a collection of strings, each with an associated probability of occurrence, the guesswork of each of them is their position in a list ordered from most likely to least likely, breaking ties arbitrarily. Guesswork is central to several…