Related papers: Information Distance Revisited
We studied the mutual information between a stimulus and a large system consisting of stochastic, statistically independent elements that respond to a stimulus. The Mutual Information (MI) of the system saturates exponentially with system…
For a bounded metric space X, we define a metric on the set of all finite subsets of X. This generalizes the sequence-subset distance introduced by Wentu Song, Kui Cai and Kees A. Schouhamer Immink to study error correcting codes for DNA…
We introduce a new form of logical relation which, in the spirit of metric relations, allows us to assign each pair of programs a quantity measuring their distance, rather than a boolean value standing for their being equivalent. The…
We establish a precise mathematical equivalence between witness-based similarity systems (REWA) and Shannon's information theory. We prove that witness overlap is mutual information, that REWA bit complexity bounds arise from channel…
Edit distance is a fundamental measure of distance between strings and has been widely studied in computer science. While the problem of estimating edit distance has been studied extensively, the equally important question of actually…
The exact common information between a set of random variables $X_1,...,X_n$ is defined as the minimum entropy of a shared random variable that allows for the exact distributive simulation of $X_1,...,X_n$. It has been established that, in…
The Discounted Least Information Theory of Entropy (DLITE) is a new information measure that quantifies the amount of entropic difference between two probability distributions. It manifests multiple critical properties both as an…
We refer to the distance between optimal solutions of integer programs and their linear relaxations as proximity. In 2018, Eisenbrand and Weismantel proved that proximity is independent of the dimension for programs in standard form. We…
This paper is concerned with factor of i.i.d. processes on the $d$-regular tree for $d \geq 3$. We study the mutual information of the values on two given vertices. If the vertices are neighbors (i.e., their distance is $1$), then a known…
Inference capabilities of machine learning (ML) systems skyrocketed in recent years, now playing a pivotal role in various aspect of society. The goal in statistical learning is to use data to obtain simple algorithms for predicting a…
R\'enyi divergence is related to R\'enyi entropy much like information divergence (also called Kullback-Leibler divergence or relative entropy) is related to Shannon's entropy, and comes up in many settings. It was introduced by R\'enyi as…
Information bottleneck is an information-theoretic principle of representation learning that aims to learn a maximally compressed representation that preserves as much information about labels as possible. Under this principle, two…
The minimum rate needed to accurately approximate a product distribution based on an unnormalized informational divergence is shown to be a mutual information. This result subsumes results of Wyner on common information and Han-Verd\'{u} on…
We revisit a fundamental problem in string matching: given a pattern of length m and a text of length n, both over an alphabet of size $\sigma$, compute the Hamming distance between the pattern and the text at every location. Several…
The information metric provides a unique notion of distance along any continuous family of physical theories, placing two theories further apart the more distinguishable they are. As such, the information metric is typically singular at…
Real-world data typically contain a large number of features that are often heterogeneous in nature, relevance, and also units of measure. When assessing the similarity between data points, one can build various distance measures using…
Extracting relevant information from data is crucial for all forms of learning. The information bottleneck (IB) method formalizes this, offering a mathematically precise and conceptually appealing framework for understanding learning…
The metric properties of the set in which random variables take their values lead to relevant probabilistic concepts. For example, the mean of a random variable is a best predictor in that it minimizes the standard Euclidean distance or…
This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The discussion emphasizes the pragmatic elements in the derivation. An epistemic notion of information is defined in…
For any $T \geq 1$, there are constants $R=R(T) \geq 1$ and $\zeta=\zeta(T)>0$ and a randomized algorithm that takes as input an integer $n$ and two strings $x,y$ of length at most $n$, and runs in time $O(n^{1+\frac{1}{T}})$ and outputs an…