Related papers: Maximum Entropy, Word-Frequency, Chinese Character…
Abbreviation is a common phenomenon across languages, especially in Chinese. In most cases, if an expression can be abbreviated, its abbreviation is used more often than its fully expanded forms, since people tend to convey information in a…
We use a multivariate central limit theorem (CLT) to study the distribution of random geometric graphs (RGGs) on the cube and torus in the high-dimensional limit with general node distributions. We find that the distribution of RGGs on the…
Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of…
Probabilistic word embeddings have shown effectiveness in capturing notions of generality and entailment, but there is very little work on doing the analogous type of investigation for sentences. In this paper we define probabilistic models…
In a language corpus, the probability that a word occurs $n$ times is often proportional to $1/n^2$. Assigning rank, $s$, to words according to their abundance, $\log s$ vs $\log n$ typically has a slope of minus one. That simple Zipf's law…
The classical problem of maximizing the Shannon entropy of a sum of independent random variables supported on a finite alphabet is considered and settled in the ternary case. Namely, the following theorem is established: if…
Using statistical thermodynamics, we derive a general expression of the stationary probability distribution for thermodynamic systems driven out of equilibrium by several thermodynamic forces. The local equilibrium is defined by imposing…
The distribution of word probabilities in the monkey model of Zipf's law is associated with two universality properties: (1) the power law exponent converges strongly to $-1$ as the alphabet size increases and the letter probabilities are…
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible constrained to match empirical data, for instance, feature expectations. We seek to generalize…
The Renyi distribution ensuring the maximum of a Renyi entropy is investigated for a particular case of a power--law Hamiltonian. Both Lagrange parameters, $\alpha$ and $\beta$ can be excluded. It is found that $\beta$ does not depend on a…
For words of length n, generated by independent geometric random variables, we consider the average value and the average position of the r-th left-to-right maximum, for fixed r and large n.
Word embeddings have recently been shown to reflect many of the pronounced societal biases (e.g., gender bias or racial bias). Existing studies are, however, limited in scope and do not investigate the consistency of biases across relevant…
In this work we explore the ability of the Google search engine to find results for random N-letter strings. These random strings, dense over the set of possible N-letter words, address the existence of typos, acronyms, and other words…
In this paper we combine statistical analysis of large text databases and simple stochastic models to explain the appearance of scaling laws in the statistics of word frequencies. Besides the sublinear scaling of the vocabulary size with…
We find the value of constants related to constraints in characterization of some known statistical distributions and then we proceed to use the idea behind maximum entropy principle to derive generalized version of this distributions using…
We study the problem of computing the probability that a given stochastic context-free grammar (SCFG), G, generates a string in a given regular language L(D) (given by a DFA, D). This basic problem has a number of applications in…
Through viewing out the literature, many generated distributions took a new special form of probability density function (PDF) in which it is written as a linear combination of n other distributions. Therefore, we define in this paper a new…
Controlled generation refers to the problem of creating text that contains stylistic or semantic attributes of interest. Many approaches reduce this problem to training a predictor of the desired attribute. For example, researchers hoping…
Here we sketch a new derivation of Zipf's law for word frequencies based on optimal coding. The structure of the derivation is reminiscent of Mandelbrot's random typing model but it has multiple advantages over random typing: (1) it starts…
Shannon Entropy is the preeminent tool for measuring the level of uncertainty (and conversely, information content) in a random variable. In the field of communications, entropy can be used to express the information content of given…