Related papers: Information Theory: A Tutorial Introduction
In many different fields of science, it is useful to characterize physical states and processes as resources. Chemistry, thermodynamics, Shannon's theory of communication channels, and the theory of quantum entanglement are prominent…
A theory of how agents can come to understand a language is presented. If understanding a sentence $\alpha$ is to associate an operator with $\alpha$ that transforms the representational state of the agent as intended by the sender, then…
Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order patterns with distinct properties across scales. Investigating how…
The field of Information Theory is founded on Claude Shannon's seminal ideas relating to entropy. Nevertheless, his well-known avoidance of meaning (Shannon, 1948) still persists to this day, so that Information Theory remains poorly…
One of the main notions of information theory is the notion of mutual information in two messages (two random variables in Shannon information theory or two binary strings in algorithmic information theory). The mutual information in $x$…
This paper proposes a reconciliation of two different theories of information. The first, originally proposed in a lesser-known work by Claude Shannon, describes how the information content of channels can be described qualitatively, but…
Information must take up space, must weigh, and its flux must be limited. Quantum limits on communication and information storage leading to these conclusions are here described. Quantum channel capacity theory is reviewed for both steady…
Molecular Communication (MC) is a communication strategy that uses molecules as carriers of information, and is widely used by biological cells. As an interdisciplinary topic, it has been studied by biologists, communication theorists and a…
In this paper, firstly, the Shannon channel capacity formula is briefly stated, and the relationship between the formula and the signal uncertainty principle is analyzed in order to prepare for deriving the formula which is able to break…
I consider the effect of a finite sample size on the entropy of a sample of independent events. I propose formula for entropy which satisfies Shannon's axioms, and which reduces to Shannon's entropy when sample size is infinite. I discuss…
Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome's plausibility. Information measures based on…
The Shannon upper bound places a limit on the error-free information transmission rate (capacity) of a noisy channel. It has stood for over sixty years, and underlies both theoretical and practical work in the telecommunications industry.…
We provide a fundamental treatment of the molecular communication channel wherein "inscribed matter" is transmitted across a spatial gap to provide reliable signaling between a sender and receiver. Inscribed matter is defined as an ensemble…
Accurately determining dependency structure is critical to discovering a system's causal organization. We recently showed that the transfer entropy fails in a key aspect of this---measuring information flow---due to its conflation of dyadic…
Fundamental limits on the controllability of physical systems are discussed in the light of information theory. It is shown that the second law of thermodynamics, when generalized to include information, sets absolute limits to the minimum…
Despite the fact that Shannon and Weaver's Mathematical Theory of Communication was published over 70 years ago, all communication systems continue to operate at the first of three levels defined in this theory: the technical level. In this…
Realistic modeling of brain involves large number of neurons. The important question is how this size affects transmission efficiency? Here, these issue is studied in terms of Shannon's Theory. Mutual Information between input and output…
At the 2023 Les Houches Summer School on Theoretical Biological Physics, several students asked for some background on information theory, and so we added a tutorial to the scheduled lectures. This is largely a transcript of that tutorial,…
"Standard" information theory says nothing about the semantic content of information. Nevertheless, applications such as evolutionary theory demand consideration of precisely this aspect of information, a need that has motivated a largely…
How much information can a transmitted physical system fundamentally communicate? We introduce the principle of quantum information causality, which states the maximum amount of quantum information that a quantum system can communicate as a…