Related papers: Symmetry in Shannon's Noiseless Coding Theorem
Our capacity to process information depends on the computational power at our disposal. Information theory captures our ability to distinguish states or communicate messages when it is unconstrained with unrivaled beauty and elegance. For…
Noiseless coding theorem for finite stationary memoryless information sources is proved by using induction on the number of source symbols and the inequality of geometric and harmonic means.
A unified combinatorial definition of the information content and entropy of different types of patterns, compatible with the traditional concepts of information and entropy, going beyond the limitations of Shannon information interpretable…
We generalize the convex duality symmetry in Gibbs' statistical ensemble formulation, between Massieu's free entropy $\Phi_{V,N} (\beta)$ and the Gibbs entropy $\varphi_{V,N}(u)$ as a function of mean internal energy $u$. The duality tells…
The Shannon lower bound is one of the few lower bounds on the rate-distortion function that holds for a large class of sources. In this paper, it is demonstrated that its gap to the rate-distortion function vanishes as the allowed…
We consider a simplified version of a solvable model by Mandal and Jarzynski, which constructively demonstrates the interplay between work extraction and the increase of the Shannon entropy of an information reservoir which is in contact…
The paper deals with the generalization of both Boltzmann entropy and distribution in the light of most-probable interpretation of statistical equilibrium. The statistical analysis of the generalized entropy and distribution leads to some…
The Shannon capacity of a graph G is the maximum asymptotic rate at which messages can be sent with zero probability of error through a noisy channel with confusability graph G. This extensively studied graph parameter disregards the fact…
Shannon information entropies in position and momentum spaces and their sum are calculated as functions of Z (Z=2-54) in atoms. Roothaan-Hartree-Fock electron wave functions are used. The universal property S=a+b lnZ is verified. In…
In this work, the proofs concerning the continuity of the disequilibrium, Shannon information and statistical complexity in the space of distributions are presented. Also, some results on the existence of Shannon information for continuous…
The generalization of Shannon's theory to include messages with given autocorrelations is presented. The analytical calculation of the channel capacity is based on the transfer matrix method of the effective 1D Hamiltonian. This bridge…
Building upon the work of Chebyshev, Shannon and Kontoyiannis, it may be demonstrated that Chebyshev's asymptotic result: \begin{equation} \ln N \sim \sum_{p \leq N} \frac{1}{p} \cdot \ln p \end{equation} has a natural information-theoretic…
There is a parallelism between Shannon information theory and algorithmic information theory. In particular, the same linear inequalities are true for Shannon entropies of tuples of random variables and Kolmogorov complexities of tuples of…
The Shannon upper bound places a limit on the error-free information transmission rate (capacity) of a noisy channel. It has stood for over sixty years, and underlies both theoretical and practical work in the telecommunications industry.…
The statistics of work done on a quantum system can be quantified by the two-point measurement scheme. We show how the Shannon entropy of the work distribution admits a general upper bound depending on the initial diagonal entropy, and a…
Strong and general entropic and geometric Heisenberg limits are obtained, for estimates of multiparameter unitary displacements in quantum metrology, such as the estimation of a magnetic field from the induced rotation of a probe state in…
Neural networks have gained importance as the machine learning models that achieve state-of-the-art performance on large-scale image classification, object detection and natural language processing tasks. In this paper, we consider noisy…
Shannon's theory of zero-error communication is re-examined in the broader setting of using one classical channel to simulate another exactly, and in the presence of various resources that are all classes of non-signalling correlations:…
It is well-known that von Neumann entropy is nonmonotonic unlike Shannon entropy (which is monotonically nondecreasing). Consequently, it is difficult to relate the entropies of the subsystems of a given quantum state. In this paper, we…
We show how generalized Gibbs-Shannon entropies can provide new insights on the statistical properties of texts. The universal distribution of word frequencies (Zipf's law) implies that the generalized entropies, computed at the word level,…