Related papers: Explaining Zipf's Law via Mental Lexicon
Long-range correlations are found in symbolic sequences from human language, music and DNA. Determining the span of correlations in dolphin whistle sequences is crucial for shedding light on their communicative complexity. Dolphin whistles…
This work proves that ranks and shares are statistically dependent on one another, based on simple combinatorics. It presents a formula for rank-share distribution and illustrates that Zipfs law, is descended from expected values of various…
The rank-size plots of a large number of different physical and socio-economic systems are usually said to follow Zipf's law, but a unique framework for the comprehension of this ubiquitous scaling law is still lacking. Here we show that a…
Tokenization is a fundamental step in natural language processing (NLP) and other sequence modeling domains, where the choice of vocabulary size significantly impacts model performance. Despite its importance, selecting an optimal…
Causal processes can give rise to distinctive distributions in the linguistic variables that they affect. Consequently, a secure understanding of a variable's distribution can hold a key to understanding the forces that have causally shaped…
Zipf's law establishes a scaling behavior for word-frequencies in large text corpora. The appearance of Zipfian properties in human language has been previously explained as an optimization problem for the interests of speakers and hearers.…
The binary many-step Markov chain with the step-like memory function is considered as a model for the analysis of rank distributions of words in stochastic symbolic dynamical systems. We prove that the envelope curve for this distribution…
We model the generation of words with independent unequal probabilities of occurrence of letters. We prove that the probability $p(r)$ of occurrence of words of rank $r$ has a power asymptotics. As distinct from the paper published earlier…
We present empirical data on frequency and pattern of misprints in citations to twelve high-profile papers. We find that the distribution of misprints, ranked by frequency of their repetition, follows Zipf's law. We propose a stochastic…
Understanding the innovation process, that is the underlying mechanisms through which novelties emerge, diffuse and trigger further novelties is undoubtedly of fundamental importance in many areas (biology, linguistics, social science and…
We introduce a simple and generic model that reproduces Zipf's law. By regarding the time evolution of the model as a random walk in the logarithmic scale, we explain theoretically why this model reproduces Zipf's law. The explanation shows…
We perform a quantitative analysis of extensive chess databases and show that the frequencies of opening moves are distributed according to a power-law with an exponent that increases linearly with the game depth, whereas the pooled…
The problem of compression in standard information theory consists of assigning codes as short as possible to numbers. Here we consider the problem of optimal coding -- under an arbitrary coding scheme -- and show that it predicts Zipf's…
A plethora of natural and socio-economic phenomena share a striking statistical regularity, that is the magnitude of elements decreases with a power law as a function of their position in a ranking of magnitude. Such regularity is known as…
We present a general approach to explain the Zipf's law of city distribution. If the simplest interaction (pairwise) is assumed, individuals tend to form cities in agreement with the well-known statistics
Background: Zipf's discovery that word frequency distributions obey a power law established parallels between biological and physical processes, and language, laying the groundwork for a complex systems perspective on human communication.…
Entity extraction is critical in the intelligent advancement across diverse domains. Nevertheless, a challenge to its effectiveness arises from the data imbalance. This paper proposes a novel approach by viewing the issue through the…
Zipf's power law is a general empirical regularity found in many natural and social systems. A recently developed theory predicts that Zipf's law corresponds to systems that are growing according to a maximally sustainable path in the…
Zipf's law for cities is probably the most famous regularity in social sciences. So much that, a hundred years of publication later, its status is not clear: is it a law of social organisation? Is it an instrument of description of city…
The rank-size regularity known as Zipf's law is one of scaling laws and frequently observed within the natural living world and in social institutions. Many scientists tried to derive the rank-size scaling relation by entropy-maximizing…