Explaining Zipf's Law via Mental Lexicon
Data Analysis, Statistics and Probability
2015-06-15 v1 Statistical Mechanics
Computation and Language
Abstract
The Zipf's law is the major regularity of statistical linguistics that served as a prototype for rank-frequency relations and scaling laws in natural sciences. Here we show that the Zipf's law -- together with its applicability for a single text and its generalizations to high and low frequencies including hapax legomena -- can be derived from assuming that the words are drawn into the text with random probabilities. Their apriori density relates, via the Bayesian statistics, to general features of the mental lexicon of the author who produced the text.
Keywords
Cite
@article{arxiv.1302.4383,
title = {Explaining Zipf's Law via Mental Lexicon},
author = {Armen E. Allahverdyan and Weibing Deng and Q. A. Wang},
journal= {arXiv preprint arXiv:1302.4383},
year = {2015}
}