Related papers: Entropy of a Zipfian Distributed Lexicon
Zipf's law implies the statistical distributions of hyperbolic type, which can describe the properties of stability and entropy loss in linguistics. We present the information theory from which follows that if the system is described by…
The mapping of lexical meanings to wordforms is a major feature of natural languages. While usage pressures might assign short words to frequent meanings (Zipf's law of abbreviation), the need for a productive and open-ended vocabulary,…
Systematic generalization remains challenging for current language models, which are known to be both sensitive to semantically similar permutations of the input and to struggle with known concepts presented in novel contexts. Although…
Preserving biodiversity and ecosystem stability is a challenge that can be pursued through modern statistical mechanics modeling. Here we introduce a variational maximum entropy-based algorithm to evaluate the entropy in a minimal ecosystem…
Making decisions freely presupposes that there is some indeterminacy in the environment and in the decision making engine. The former is reflected on the behavioral changes due to communicating: few changes indicate rigid environments;…
The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be…
We use the formalism of 'Maximum Principle of Shannon's Entropy' to derive the general power law distribution function, using what seems to be a reasonable physical assumption, namely, the demand of a constant mean "internal order"…
Within the task of collaborative filtering two challenges for computing conditional probabilities exist. First, the amount of training data available is typically sparse with respect to the size of the domain. Thus, support for higher-order…
We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our…
Organizations generate vast amounts of interconnected content across various platforms. While language models enable sophisticated reasoning for use in business applications, retrieving and contextualizing information from organizational…
The word-frequency distribution provides the fundamental building blocks that generate discourse in language. It is well known, from empirical evidence, that the word-frequency distribution of almost any text is described by Zipf's law, at…
We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the…
Under certain conditions, the rate of increase of the statistical entropy of a simple, fully chaotic, conservative system is known to be given by a single number, characteristic of this system, the Kolmogorov-Sinai entropy rate. This…
A mathematical interpretation of the usual definition of entropy (for a discrete probability distribution or a trace 1 positive operator) is given. This formulation makes some properties of entropy immediate.
We review recent progress in understanding the meaning of mutual information in natural language. Let us define words in a text as strings that occur sufficiently often. In a few previous papers, we have shown that a power-law distribution…
We consider relaxation of an isolated system to the equilibrium using detailed balance condition and Onsager's fluctuation approximation. There is a small deviation from the equilibrium in two parameters. For this system, explicit…
This paper aims to show how some popular topics on social networks can be used to predict online newspaper views, related to the topics. Newspapers site and many social networks, become a good source of data to analyse and explain complex…
We study the entropy of Chinese and English texts, based on characters in case of Chinese texts and based on words for both languages. Significant differences are found between the languages and between different personal styles of debating…
This paper proposes a novel entropy encoding technique for lossless data compression. Representing a message string by its lexicographic index in the permutations of its symbols results in a compressed version matching Shannon entropy of…
The Zipf's law establishes that if the words of a (large) text are ordered by decreasing frequency, the frequency versus the rank decreases as a power law with exponent close to $-1$. Previous work has stressed that this pattern arises from…