Related papers: Maximum Entropy, Word-Frequency, Chinese Character…
Text classification has become indispensable due to the rapid increase of text in digital form. Over the past three decades, efforts have been made to approach this task using various learning algorithms and statistical models based on…
Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty…
The weak law of large numbers implies that, under mild assumptions on the source, the Renyi entropy per produced symbol converges (in probability) towards the Shannon entropy rate. This paper quantifies the speed of this convergence for…
Recent deep learning based approaches have achieved great success on handwriting recognition. Chinese characters are among the most widely adopted writing systems in the world. Previous research has mainly focused on recognizing handwritten…
Subshifts of deterministic substitutions are ubiquitous objects in dynamical systems and aperiodic order (the mathematical theory of quasicrystals). Two of their most striking features are that they have low complexity (zero topological…
The binary many-step Markov chain with the step-like memory function is considered as a model for the analysis of rank distributions of words in stochastic symbolic dynamical systems. We prove that the envelope curve for this distribution…
Zipf's law is the main regularity of quantitative linguistics. Despite of many works devoted to foundations of this law, it is still unclear whether it is only a statistical regularity, or it has deeper relations with information-carrying…
In this paper we consider the density of maximal order elements in $\mathrm{GL}_n(q)$. Fixing any of the rank $n$ of the group, the characteristic $p$ or the degree $r$ of the extension of the underlying field $\mathbb{F}_q$ of size…
Stopwords are words that are not very informative to the content or the meaning of a language text. Most stopwords are function words but can also be common verbs, adjectives and adverbs. In contrast to the well known Zipf's law for…
Given a random text over a finite alphabet, we study the frequencies at which fixed-length words occur as subsequences. As the data size grows, the joint distribution of word counts exhibits a rich asymptotic structure. We investigate all…
The need to condition distributional properties such as expectation, variance, and entropy arises in algorithmic fairness, model simplification, robustness and many other areas. At face value however, distributional properties are not…
The generalised random Fibonacci chain is a stochastic extension of the classical Fibonacci substitution and is defined as the rule mapping $0\mapsto 1$ and $1 \mapsto 1^i01^{m-i}$ with probability $p_i$, where $p_i\geq 0$ with…
We consider distributions of ordered random vectors with given one-dimensional marginal distributions. We give an elementary necessary and sufficient condition for the existence of such a distribution with finite entropy. In this case, we…
Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distribution requires enormous amounts of data. Representations of rare…
Recent works have demonstrated success in controlling sentence attributes ($e.g.$, sentiment) and structure ($e.g.$, syntactic structure) based on the diffusion language model. A key component that drives theimpressive performance for…
We study the entanglement entropy of a random tensor network (RTN) using tools from free probability theory. Random tensor networks are simple toy models that help the understanding of the entanglement behavior of a boundary region in the…
Expressive text encoders such as RNNs and Transformer Networks have been at the center of NLP models in recent work. Most of the effort has focused on sentence-level tasks, capturing the dependencies between words in a single sentence, or…
Recent studies have shown that as Transformer-based language models become larger and are trained on very large amounts of data, the fit of their surprisal estimates to naturalistic human reading times degrades. The current work presents a…
Text classification, as the task consisting in assigning categories to textual instances, is a very common task in information science. Methods learning distributed representations of words, such as word embeddings, have become popular in…
On a compact group the Haar probability measure plays the role of uniform distribution. The entropy and rate distortion theory for this uniform distribution is studied. New results and simplified proofs on convergence of convolutions on…