Related papers: Scaling Laws in Human Language
A set of data with positive values follows a Pareto distribution if the log-log plot of value versus rank is approximately a straight line. A Pareto distribution satisfies Zipf's law if the log-log plot has a slope of -1. Since many types…
The nature of the quantitative distribution of the 64 DNA codons in the human genome has been an issue of debate for over a decade. Some groups have proposed that the quantitative distribution of the DNA codons ordered as a rank-frequency…
This paper introduces new methods based on exponential families for modeling the correlations between words in text and speech. While previous work assumed the effects of word co-occurrence statistics to be constant over a window of several…
We study the frequency distribution of family names. From a common data base, we count the number of people who share the same family name. This is the size of the family. We find that (i) the total number of different family names in a…
The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be…
The length of coding sequence series in microbial genomes were regarded as a fluctuating system and characterized by the methods of statistical physics. The distribution and the correlatin properties of 50 genomes including bacteria and…
When the probability of measuring a particular value of some quantity varies inversely as a power of that value, the quantity is said to follow a power law, also known variously as Zipf's law or the Pareto distribution. Power laws appear…
The availability of large datasets requires an improved view on statistical laws in complex systems, such as Zipf's law of word frequencies, the Gutenberg-Richter law of earthquake magnitudes, or scale-free degree distribution in networks.…
Sentence formation is a highly structured, history-dependent, and sample-space reducing (SSR) process. While the first word in a sentence can be chosen from the entire vocabulary, typically, the freedom of choosing subsequent words gets…
Stopwords are words that are not very informative to the content or the meaning of a language text. Most stopwords are function words but can also be common verbs, adjectives and adverbs. In contrast to the well known Zipf's law for…
The diffusion of ideas and language in society has conventionally been described by S-shaped models, such as the logistic curve. However, the role of sub-exponential growth -- a slower-than-exponential pattern known in epidemiology -- has…
A family of information theoretic models of communication was introduced more than a decade ago to explain the origins of Zipf's law for word frequencies. The family is a based on a combination of two information theoretic principles:…
We summarize a book under publication with his title written by the three present authors, on the theory of Zipf's law, and more generally of power laws, driven by the mechanism of proportional growth. The preprint is available upon request…
English words and the outputs of many other natural processes are well-known to follow a Zipf distribution. Yet this thoroughly-established property has never been shown to help compress or predict these important processes. We show that…
We show how Zipf's Law can be used to scale up language modeling (LM) to take advantage of more training data and more GPUs. LM plays a key role in many important natural language applications such as speech recognition and machine…
The analysis of thousands of time series in different languages reveals that word usage presents oscillations with a prevalence of 16-year cycles, mounted on slowly varying trends. These components carry different information: while similar…
The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its…
Taylor's law describes the fluctuation characteristics underlying a system in which the variance of an event within a time span grows by a power law with respect to the mean. Although Taylor's law has been applied in many natural and social…
Zipf's power law is a general empirical regularity found in many natural and social systems. A recently developed theory predicts that Zipf's law corresponds to systems that are growing according to a maximally sustainable path in the…
Bayesian modelling and statistical text analysis rely on informed probability priors to encourage good solutions. This paper empirically analyses whether text in medical discharge reports follow Zipf's law, a commonly assumed statistical…