Related papers: Information Bottleneck on General Alphabets

We adapt arguments concerning information-theoretic convergence in the Central Limit Theorem to the case of dependent random variables under Rosenblatt mixing conditions. The key is to work with random variables perturbed by the addition of…

In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior probing work, most of which investigates how much information a…

Exploiting recent developments in information theory, we propose, illustrate, and validate a principled information-theoretic algorithm for module discovery and resulting measure of network modularity. This measure is an order parameter (a…

We prove a quenched functional central limit theorem (quenched FCLT) for the sums of a random field (r.f.) along a Z d-random walk in different frameworks: probabilistic (when the r.f. is i.i.d. or a moving average of i.i.d. random…

Let \mathcal{F}_k denote the family of 2-edge-colored complete graphs on 2k vertices in which one color forms either a clique of order k or two disjoint cliques of order k. Bollob\'as conjectured that for every \epsilon>0 and positive…

In many complex systems, we observe that `interesting behaviour' is often the consequence of a system exploiting the existence of an Information Bottleneck (IB). These bottlenecks can occur at different scales, between individuals or…

We introduce fundamental bounds on achievable cumulative rate distribution functions (CRDF) to characterize a sequential encoding process that ensures lossless or lossy reconstruction subject to an average distortion criterion using a…

We investigate which infinite binary sequences (reals) are effectively random with respect to some continuous (i.e., non-atomic) probability measure. We prove that for every n, all but countably many reals are n-random for such a measure,…

We prove the Courtade-Kumar conjecture, for certain classes of $n$-dimensional Boolean functions, $\forall n\geq 2$ and for all values of the error probability of the binary symmetric channel, $\forall 0 \leq p \leq \frac{1}{2}$. Let…

Let $(\mathbf X, \mathbf Y)$ denote $n$ independent, identically distributed copies of two arbitrarily correlated Rademacher random variables $(X, Y)$. We prove that the inequality $I(f(\mathbf X); g(\mathbf Y)) \le I(X; Y)$ holds for any…

Decisions of complex language understanding models can be rationalized by limiting their inputs to a relevant subsequence of the original text. A rationale should be as concise as possible without significantly degrading task performance,…

Let $X=(X_1,\ldots,X_n)$ be a vector of i.i.d. random variables where $X_i$'s take values over $\mathbb{N}$. The purpose of this paper is to study the number of weakly increasing subsequences of $X$ of a given length $k$, and the number of…

In this paper, we prove that the Fourier entropy of an $n$-dimensional boolean function $f$ can be upper-bounded by $O(I(f)+ \sum\limits_{k\in[n]}I_k(f)\log \frac{1}{I_k(f)})$, where $I(f)$ is its total influence and $I_k(f)$ is the…

We generalize the feasible interpolation theorem for semantic derivations from K.(1997) by allowing randomized protocols (protocols in the sense of K.(1997). We also introduce an extension of the monotone circuit model, monotone circuits…

We introduce the matrix-based Renyi's $\alpha$-order entropy functional to parameterize Tishby et al. information bottleneck (IB) principle with a neural network. We term our methodology Deep Deterministic Information Bottleneck (DIB), as…

This manuscript includes some classical results we select apart from the new results we've found on the Analysis of Boolean Functions and Fourier-Entropy-Influence conjecture. We try to ensure the self-completeness of this work so that…

We prove two new results about the randomized query complexity of composed functions. First, we show that the randomized composition conjecture is false: there are families of partial Boolean functions $f$ and $g$ such that $R(f\circ g)\ll…

In this paper, we are interested in the number of fixed points of functions $f:A^n\to A^n$ over a finite alphabet $A$ defined on a given signed digraph $D$. We first use techniques from network coding to derive some lower bounds on the…

Information bottleneck is an information-theoretic principle of representation learning that aims to learn a maximally compressed representation that preserves as much information about labels as possible. Under this principle, two…

By the Lindeberg-L\'evy central limit theorem, standardized partial sums of a sequence of mutually independent and identically distributed random variables converge in law to the standard normal distribution. It is known that mutual…