Related papers: Shannon Information and Kolmogorov Complexity
We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the…
We show that a proper expression of the uncertainty relation for a pair of canonically-conjugate continuous variables relies on entropy power, a standard notion in Shannon information theory for real-valued signals. The resulting…
The Radon-Nikodym theorem plays a significant role in the definition of Shannon entropy, f-divergences, and other basic quantities in information theory. The existence of Radon Nikodym derivates appear in many text books in measure theory…
From the output produced by a memoryless deletion channel with a uniformly random input of known length $n$, one obtains a posterior distribution on the channel input. The difference between the Shannon entropy of this distribution and that…
We study the relations between the recently proposed machine-independent quantum complexity of P. Gacs~\cite{Gacs} and the entropy of classical and quantum systems. On one hand, by restricting Gacs complexity to ergodic classical dynamical…
In this paper from communication channel coding perspective we are able to present both a theoretical and practical discussion of AI's uncertainty, capacity and evolution for pattern classification based on the classical Rademacher…
The information in an individual finite object (like a binary string) is commonly measured by its Kolmogorov complexity. One can divide that information into two parts: the information accounting for the useful regularity present in the…
A possible link between Holography and Information Theory is presented. Using the relation between the Shannon and Boltzmann formulas Holography can be seen as the best encoding scheme.
Information theory is a mathematical theory of learning with deep connections with topics as diverse as artificial intelligence, statistical physics, and biological evolution. Many primers on information theory paint a broad picture with…
Shannon's rate-distortion theory treats source symbols as unstructured labels. When the source is a knowledge base equipped with a logical proof system, a natural fidelity criterion is closure fidelity: a reconstruction is acceptable if it…
I proposed rate tolerance and discussed its relation to rate distortion in my book "A Generalized Information Theory" published in 1993. Recently, I examined the structure function and the complexity distortion based on Kolmogorov's…
Accurately matching visual and textual data in cross-modal retrieval has been widely studied in the multimedia community. To address these challenges posited by the heterogeneity gap and the semantic gap, we propose integrating Shannon…
Pseudoentropy characterizations provide a quantitatively precise demonstration of the close relationship between computational hardness and computational randomness. We prove a unified pseudoentropy characterization that generalizes and…
Alice and Bob are given two correlated n-bit strings x_1 and, respectively, x_2, which they want to losslessly compress and send to Zack. They can either collaborate by sharing their strings, or work separately. We show that there is no…
Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated…
Fundamental relations between information and estimation have been established in the literature for the continuous-time Gaussian and Poisson channels, in a long line of work starting from the classical representation theorems by Duncan and…
We show that an information-theoretic property of Shannon's entropy power, known as concavity of entropy power, can be fruitfully employed to prove inequalities in sharp form. In particular, the concavity of entropy power implies the…
The Kolmogorov complexity of a string is the length of its shortest description. We define a second quantised Kolmogorov complexity where the length of a description is defined to be the average length of its superposition. We discuss this…
For noncomposite systems in classical and quantum domains, we obtain new inequalities such as the subadditivity and strong subadditivity conditions for Shannon entropies and information determined by the probability distributions and for…
The field of Information Theory is founded on Claude Shannon's seminal ideas relating to entropy. Nevertheless, his well-known avoidance of meaning (Shannon, 1948) still persists to this day, so that Information Theory remains poorly…