Related papers: Information Bottleneck on General Alphabets
Information bottleneck (IB) is a paradigm to extract information in one target random variable from another relevant random variable, which has aroused great interest due to its potential to explain deep neural networks in terms of…
The concept of Shannon entropy of random variables was generalized to measurable functions in general, and to simple functions with finite values in particular. It is shown that the information measure of a function is related to the time…
This monograph presents a unified treatment of single- and multi-user problems in Shannon's information theory where we depart from the requirement that the error probability decays asymptotically in the blocklength. Instead, the error…
This paper fills a gap in our understanding of the interaction between information and computation. It unifies other approaches to measuring information like Kolmogorov complexity and Shannon information. We define a theory about…
In this paper, we take a unified approach for network information theory and prove a coding theorem, which can recover most of the achievability results in network information theory that are based on random coding. The final single-letter…
The following learning problem arises naturally in various applications: Given a finite sample from a categorical or count time series, can we learn a function of the sample that (nearly) maximizes the probability of correctly guessing the…
In terms of the Dirac representation of sample mean and the weak convergence of empirical distributions that holds almost surely, we construct a new proof for a strong law of large numbers of Kolmogorov's type with i.i.d. random variables…
We study the information leakage to a guessing adversary in zero-error source coding. The source coding problem is defined by a confusion graph capturing the distinguishability between source symbols. The information leakage is measured by…
The multivariate covering lemma states that given a collection of $k$ codebooks, each of sufficiently large cardinality and independently generated according to one of the marginals of a joint distribution, one can always choose one…
The Central Limit Theorem (CLT) is one of the most fundamental results in statistics. It states that the standardized sample mean of a sequence of $n$ mutually independent and identically distributed random variables with finite first and…
Link prediction (LP) is a fundamental task in graph representation learning, with numerous applications in diverse domains. However, the generalizability of LP models is often compromised due to the presence of noisy or spurious information…
We expand upon the notion of bottlenecking introduced in our earlier work, characterizing a spectrum of graphs and showing that this naturally extends to a concept of coarse bottlenecking. We show how the notion of bottlenecking provides a…
It has been shown that Information-Disturbance theorem can play an important role in security proof of quantum cryptography. The theorem is by itself interesting since it can be regarded as an information theoretic version of uncertainty…
Fake news poses global risks by influencing elections and spreading misinformation, making detection critical. Existing NLP and supervised Machine Learning methods perform well under cross-validation but struggle to generalise across…
The problem of guessing a random string is revisited. A close relation between guessing and compression is first established. Then it is shown that if the sequence of distributions of the information spectrum satisfies the large deviation…
We use Hidden Markov Models to motivate a quantitative compositional semantics for noninterference-based security with iteration, including a refinement- or "implements" relation that compares two programs with respect to their information…
We study the distribution of partial sums of Rademacher random multiplicative functions $(f(n))_n$ evaluated at polynomial arguments. We show that for a polynomial $P\in \mathbb Z[x]$ that is a product of at least two distinct linear…
A general approach is established for deriving one-shot performance bounds for information-theoretic problems on general alphabets beyond countable alphabets. It is mainly based on the quantization idea and a novel form of "likelihood…
A finite form of de Finetti's representation theorem is established using elementary information-theoretic tools: The distribution of the first $k$ random variables in an exchangeable binary vector of length $n\geq k$ is close to a mixture…
For a set $X\sbst\R$, let $B(X)\sbst\R^X$ denote the space of Borel real-valued functions on $X$, with the topology inherited from the Tychonoff product $\R^X$. Assume that for each countable $A\sbst B(X)$, each $f$ in the closure of $A$ is…