Related papers: Breaking through the classical Shannon entropy lim…
We introduce the (private) entropy of a directed graph (in a new network coding sense) as well as a number of related concepts. We show that the entropy of a directed graph is identical to its guessing number and can be bounded from below…
Probability theory is fundamental for modeling uncertainty, with traditional probabilities being real and non-negative. Complex probability extends this concept by allowing complex-valued probabilities, opening new avenues for analysis in…
"Standard" information theory says nothing about the semantic content of information. Nevertheless, applications such as evolutionary theory demand consideration of precisely this aspect of information, a need that has motivated a largely…
Semantic communication systems often use an end-to-end neural network to map input data into continuous symbols. These symbols, which are essentially neural network features, usually have fixed dimensions and heavy-tailed distributions.…
The tension between deduction and induction is perhaps the most fundamental issue in areas such as philosophy, cognition and artificial intelligence (AI). The deduction camp concerns itself with questions about the expressiveness of formal…
We provide a simple physical interpretation, in the context of the second law of thermodynamics, to the information inequality (a.k.a. the Gibbs' inequality, which is also equivalent to the log-sum inequality), asserting that the relative…
Synapses are information efficient in the sense that their natural conductance values convey as many bits per Joule as possible, but efficiency falls rapidly if the conductance is forced to deviate from its natural value (Harris et al,…
Learning disentangled representations of natural language is essential for many NLP tasks, e.g., conditional text generation, style transfer, personalized dialogue systems, etc. Similar problems have been studied extensively for other forms…
In many complex systems, whether biological or artificial, the thermodynamic costs of communication among their components are large. These systems also tend to split information transmitted between any two components across multiple…
A probabilistic source is defined as the set of infinite words (over a given denumerable alphabet) endowed with a probability $\mu$. The paper deals with general binary sources where the distribution of any symbol (0 or 1) may depend on an…
Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated…
We introduce information-theoretic erasure based on Shannon's binary channel formula. It is pointed out that this type of erasure is a natural energy-dissipation-free way in which information is lost in double-potential-well memories, and…
Semantic communications represent a significant breakthrough with respect to the current communication paradigm, as they focus on recovering the meaning behind the transmitted sequence of symbols, rather than the symbols themselves. In…
An information theory description of finite systems explicitly evolving in time is presented. We impose a MaxEnt variational principle on the Shannon entropy at a given time while the constraints are set at a former time. The resulting…
Shannon's theory of information was built on the assumption that the information carriers were classical systems. Its quantum counterpart, quantum Shannon theory, explores the new possibilities arising when the information carriers are…
Shannon entropy is the most crucial foundation of Information Theory, which has been proven to be effective in many fields such as communications. Renyi entropy and Chernoff information are other two popular measures of information with…
We characterize information as risk reduction between knowledge states represented by partitions of the underlying probability space. Entropy corresponds to risk reduction from no (or partial) knowledge to full knowledge about a random…
This paper proposes a reconciliation of two different theories of information. The first, originally proposed in a lesser-known work by Claude Shannon, describes how the information content of channels can be described qualitatively, but…
Large language models (LLMs) are increasingly used in situations where human values are at stake, such as decision-making tasks that involve reasoning when performed by humans. We investigate the so-called reasoning capabilities of LLMs…
Classical and quantum information theory are simply explained. To be more specific it is clarified why Shannon entropy is used as measure of classical information and after a brief review of quantum mechanics it is possible to demonstrate…