Related papers: Entropy of a Zipfian Distributed Lexicon
The maximum entropy method has recently been successfully introduced to a variety of natural language applications. In each of these applications, however, the power of the maximum entropy method is achieved at the cost of a considerable…
We checked that the distribution of words in text should uniform, which gives Heaps' law as natural result, that is, the number of types of words can be expressed as a power law of the number of tokens within text. We developed a…
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality: It has often been observed that mode-seeking decoding methods, i.e., those that produce high-probability text…
Nature is full of random networks of complex topology describing such apparently disparate systems as biological, economical or informatical ones. Their most characteristic feature is the apparent scale-free character of interconnections…
We construct the generalized entropy optimized by a given arbitrary statistical distribution with a finite linear expectation value of a random quantity of interest. This offers, via the maximum entropy principle, a unified basis for a…
Reliable evaluation protocols are of utmost importance for reproducible NLP research. In this work, we show that sometimes neither metric nor conventional human evaluation is sufficient to draw conclusions about system performance. Using…
The phenomenon of entropy concentration provides strong support for the maximum entropy method, MaxEnt, for inferring a probability vector from information in the form of constraints. Here we extend this phenomenon, in a discrete setting,…
In this paper we review various information-theoretic characterizations of the approach to equilibrium in biological systems. The replicator equation, evolutionary game theory, Markov processes and chemical reaction networks all describe…
Zipf's law is a fundamental paradigm in the statistics of written and spoken natural language as well as in other communication systems. We raise the question of the elementary units for which Zipf's law should hold in the most natural way,…
Statistical regularities in human language have fascinated researchers for decades, suggesting deep underlying principles governing its evolution and information structuring for efficient communication. While Zipf's Law describes the…
Entropies must correspond to mean values for them to be measurable. The Shannon entropy corresponds to the weighted arithmetic mean, whereas the Renyi entropy corresponds to the exponential mean. These means refer to code lengths, which are…
Machine-type communications (MTC) are crucial in the evolution of mobile communication systems. Within this context, we distinguish the so-called massive MTC (mMTC), where a large number of devices coexist in the same geographical area. In…
We investigate the consequences of information exchange between a system and a measurement-feedback apparatus that cools the system below the environmental temperature. A quantitative relationship between entropy pumping and information…
As is the case of many signals produced by complex systems, language presents a statistical structure that is balanced between order and disorder. Here we review and extend recent results from quantitative characterisations of the degree of…
The distribution of frequency counts of distinct words by length in a language's vocabulary will be analyzed using two methods. The first, will look at the empirical distributions of several languages and derive a distribution that…
We quantify a social organization's potentiality, that is its ability to attain different configurations. The organization is represented as a network in which nodes correspond to individuals and (multi-)edges to their multiple…
Zipf's law on word frequency is observed in English, French, Spanish, Italian, and so on, yet it does not hold for Chinese, Japanese or Korean characters. A model for writing process is proposed to explain the above difference, which takes…
We show that statistical criticality, i.e. the occurrence of power law frequency distributions, arises in samples that are maximally informative about the underlying generating process. In order to reach this conclusion, we first identify…
In this paper we study the Exponentiated Hypoexponential Distribution with different parameters. The distribution added a parameter to the n parameters of the Hypoexponenial distribution. We first derive a closed expression of the…
Fluctuation theorems impose fundamental bounds in the statistics of the entropy production, with the second law of thermodynamics being the most famous. Using information theory, we quantify the information of entropy production and find an…