Related papers: Entropy of a Zipfian Distributed Lexicon
Zipf's law describes the empirical size distribution of the components of many systems in natural and social sciences and humanities. We show, by solving a statistical model, that Zipf's law co-occurs with the maximization of the diversity…
In Shannon's seminal paper, entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its…
Calculating the Shannon entropy for symbolic sequences has been widely considered in many fields. For descriptive statistical problems such as estimating the N-gram entropy of English language text, a common approach is to use as much data…
Speech is a fundamental means of communication that can be seen to provide two channels for transmitting information: the lexical channel of which words are said, and the non-lexical channel of how they are spoken. Both channels shape…
A family of information theoretic models of communication was introduced more than a decade ago to explain the origins of Zipf's law for word frequencies. The family is a based on a combination of two information theoretic principles:…
Motivated by the approach of random linear codes, a new distance in the vector space over a finite field is defined as the logarithm of the "surface area" of a Hamming ball with radius being the corresponding Hamming distance. It is named…
The expression for entropy sometimes appears mysterious - as it often is asserted without justification. This short manuscript contains a discussion of the underlying assumptions behind entropy as well as simple derivation of this…
The postulates of thermodynamics were originally formulated for macroscopic systems. They lead to the definition of the entropy, which, for a homogeneous system, is a homogeneous function of order one in the extensive variables and is…
Maximum-entropy distributions are shown to appear in the probability calculus as approximations of a model by exchangeability or a model by sufficiency, the former model being preferable. The implications of this fact are discussed,…
A general upper bound for topological entropy of switched nonlinear systems is constructed, using an asymptotic average of upper limits of the matrix measures of Jacobian matrices of strongly persistent individual modes, weighted by their…
We propose a new method for the calculation of the statistical properties, as e.g. the entropy, of unknown generators of symbolic sequences. The probability distribution $p(k)$ of the elements $k$ of a population can be approximated by the…
Lyapunov exponents describe the asymptotic behavior of the singular values of large products of random matrices. A direct computation of these exponents is however often infeasible. By establishing a link between Lyapunov exponents and an…
During a spontaneous change, a macroscopic physical system will evolve towards a macro-state with more realizations. This observation is at the basis of the Statistical Mechanical version of the Second Law of Thermodynamics, and it provides…
The need for improved functionalities is driving the search for more complicated multi-component materials. Despite the factorially increasing composition space, ordered compounds with 4 or more species are rare. Here, we unveil the…
The word-frequency distribution of a text written by an author is well accounted for by a maximum entropy distribution, the RGF (random group formation)-prediction. The RGF-distribution is completely determined by the a priori values of the…
There are different ways of measuring diversity in complex systems. In particular, in language, lexical diversity is characterized in terms of the type-token ratio and the word entropy. We here investigate both diversity metrics in six…
For a system moving away from equilibrium, we express the entropy production via a two-point correlation function for any time and any distance from equilibrium. The long-time limit gives the sum of the Lyapunov exponents for a general…
Over the last few years, machine learning unlocked previously infeasible features for compression, such as providing guarantees for users' privacy or tailoring compression to specific data statistics (e.g., satellite images or audio…
We discuss how the method of maximum entropy, MaxEnt, can be extended beyond its original scope, as a rule to assign a probability distribution, to a full-fledged method for inductive inference. The main concept is the (relative) entropy…
We propose a new method for the calculation of the statistical properties, as e.g. the entropy, of unknown generators of symbolic sequences. The probability distribution p(k) of the elements k of a population can be approximated by the…