Related papers: Understanding Shannon's Entropy metric for Informa…
Entropy is a measure of heterogeneity widely used in applied sciences, often when data are collected over space. Recently, a number of approaches has been proposed to include spatial information in entropy. The aim of entropy is to…
The entropy is a measure of uncertainty that plays a central role in information theory. When the distribution of the data is unknown, an estimate of the entropy needs be obtained from the data sample itself. We propose a semi-parametric…
In the contemporary era, the importance of information is undisputed, but there has never been a common understanding of information, nor a unanimous conclusion to the researches on information metrics. Based on the previous studies, this…
Estimating the Shannon information associated with individual neurons is a non-trivial problem. Three key methods used to estimate the mutual information between neuron inputs and outputs are described, and a list of further readings is…
We address the problem of quantifying the information content of a source for an arbitrary information theory, where the information content is defined in terms of the asymptotic achievable compression rate. The functions that solve this…
Entropies must correspond to mean values for them to be measurable. The Shannon entropy corresponds to the weighted arithmetic mean, whereas the Renyi entropy corresponds to the exponential mean. These means refer to code lengths, which are…
Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as…
We consider the problem of finite sample corrections for entropy estimation. New estimates of the Shannon entropy are proposed and their systematic error (the bias) is computed analytically. We find that our results cover correction…
Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these…
In Shannon's seminal paper, entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its…
The entropy region is a fundamental object in information theory. An outer bound for the entropy region is defined by a minimal set of Shannon-type inequalities called elemental inequalities also referred to as the Shannon region. This…
We determine the amount of information contained in a time series of price returns at a given time scale, by using a widespread tool of the information theory, namely the Shannon entropy, applied to a symbolic representation of this time…
We start with reviewing the origin of the idea that entropy and the Second Law are associated with the Arrow of Time. We then introduced a new definition of entropy based on Shannons Measure of Information, SMI. The SMI may be defined on…
We provide a stochastic extension of the Baez-Fritz-Leinster characterization of the Shannon information loss associated with a measure-preserving function. This recovers the conditional entropy and a closely related information-theoretic…
We present a quantum information theory that allows for a consistent description of entanglement. It parallels classical (Shannon) information theory but is based entirely on density matrices (rather than probability distributions) for the…
We develop a general formalism for representing and understanding structure in complex systems. In our view, structure is the totality of relationships among a system's components, and these relationships can be quantified using information…
In many complex systems, whether biological or artificial, the thermodynamic costs of communication among their components are large. These systems also tend to split information transmitted between any two components across multiple…
Many of the traditional results in information theory, such as the channel coding theorem or the source coding theorem, are restricted to scenarios where the underlying resources are independent and identically distributed (i.i.d.) over a…
The problems of causality, modeling, and control for chaotic, high-dimensional dynamical systems are formulated in the language of information theory. The central quantity of interest is the Shannon entropy, which measures the amount of…
The form invariance of pseudoadditivity is shown to determine the structure of nonextensive entropies. Nonextensive entropy is defined as the appropriate expectation value of nonextensive information content, similar to the definition of…