Related papers: A Crucial Parameter for Rank-Frequency Relation in…
Let $f (\cdot)$ be the absolute frequency of words and $r$ be the rank of words in decreasing order of frequency, then the following function can fit the rank-frequency relation \[ f (r;s,t) = \left(\frac{r_{\tt max}}{r}\right)^{1-s}…
The Chapter starts with introductory information about quantitative linguistics notions, like rank--frequency dependence, Zipf's law, frequency spectra, etc. Similarities in distributions of words in texts with level occupation in quantum…
A sequence of real numbers $\{x_{n}\}_{n\in \mathbb{N}}$ is said to be $\alpha \beta$-statistically convergent of order $\gamma$ (where $0<\gamma\leq 1$) to a real number $x$ \cite{a} if for every $\delta>0,$ $$\underset{n\rightarrow…
Language Models (LMs) have shown promising performance in natural language generation. However, as LMs often generate incorrect or hallucinated responses, it is crucial to correctly quantify their uncertainty in responding to given inputs.…
Zipf's law predicts a power-law relationship between word rank and frequency in language communication systems and has been widely reported in a variety of natural language processing applications. However, the emergence of natural language…
We propose a stochastic model for the number of different words in a given database which incorporates the dependence on the database size and historical changes. The main feature of our model is the existence of two different classes of…
Language identification is a critical component of language processing pipelines (Jauhiainen et al.,2019) and is not a solved problem in real-world settings. We present a lightweight and effective language identifier that is robust to…
The recent dramatic increase in online data availability has allowed researchers to explore human culture with unprecedented detail, such as the growth and diversification of language. In particular, it provides statistical tools to explore…
We study rank-frequency relations for phonemes, the minimal units that still relate to linguistic meaning. We show that these relations can be described by the Dirichlet distribution, a direct analogue of the ideal-gas model in statistical…
This paper presents the Nataf-Beta Random Field Classifier, a discriminative approach that extends the applicability of the Beta conjugate prior to classification problems. The approach's key feature is to model the probability of a class…
Given an input sequence (or prefix), modern language models often assign high probabilities to output sequences that are repetitive, incoherent, or irrelevant to the prefix; as such, model-generated text also contains such artifacts. To…
We present the new empirical parameter $f_c$, the most probable usage frequency of a word in a language, computed via the distribution of documents over frequency $x$ of the word. This parameter allows for filtering the core lexicon of a…
In this work we derive the covariant and gauge invariant perturbation equations in general theories of $f(R)$ gravity in the Palatini formalism to linear order and calculate the cosmic microwave background (CMB) and matter power spectra for…
The Random Language Model, proposed as a simple model of human languages, is defined by the averaged model of a probabilistic context-free grammar. This grammar expresses the process of sentence generation as a tree graph with nodes having…
Recent studies show that Generative Relevance Feedback (GRF), using text generated by Large Language Models (LLMs), can enhance the effectiveness of query expansion. However, LLMs can generate irrelevant information that harms retrieval…
Let ${\bf L}$ be the unit exponential random variable and ${\bf Z}_\alpha$ the standard positive $\alpha$-stable random variable. We prove that $\{(1-\alpha) \alpha^{\gamma_\alpha} {\bf Z}_\alpha^{-\gamma_\alpha}, 0< \alpha <1\}$ is…
f(R) gravity is thought to be an alternative to dark energy which can explain the acceleration of the universe. It has been tested by different observations including type Ia supernovae (SNIa), the cosmic microwave background (CMB), the…
Why does Zipf's law give a good description of data from seemingly completely unrelated phenomena? Here it is argued that the reason is that they can all be described as outcomes of a ubiquitous random group division: the elements can be…
We study a deliberately simple, fully non-linguistic model of text: a sequence of independent draws from a finite alphabet of letters plus a single space symbol. A word is defined as a maximal block of non-space symbols. Within this…
The availability of large linguistic data sets enables data-driven approaches to study linguistic change. The Google Books corpus unigram frequency data set is used to investigate the word rank dynamics in eight languages. We observed the…