Related papers: Maximal Repetition and Zero Entropy Rate
Reinforcement learning (RL) has traditionally been understood from an episodic perspective; the concept of non-episodic RL, where there is no restart and therefore no reliable recovery, remains elusive. A fundamental question in…
In this paper, we consider the problem of variable-length source coding allowing errors. The exponential moment of the codeword length is analyzed in the non-asymptotic regime and in the asymptotic regime. Our results show that the smooth…
Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to…
For a text given in advance, the substring minimal suffix queries ask to determine the lexicographically minimal non-empty suffix of a substring specified by the location of its occurrence in the text. We develop a data structure answering…
We study the space requirements of a sorting algorithm where only items that at the end will be adjacent are kept together. This is equivalent to the following combinatorial problem: Consider a string of fixed length n that starts as a…
We improve the best known upper bound on the length of the shortest reset words of synchronizing automata. The new bound is slightly better than $114 n^3 / 685 + O(n^2)$. The \v{C}ern\'y conjecture states that $(n-1)^2$ is an upper bound.…
The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the…
Monotone processes, just like martingales, can often be recovered from their final values. Examples include running maxima of supermartingales, as well as running maxima, local times, and various integral functionals of sticky processes…
This paper presents new lower and upper bounds for the optimal compression of binary prefix codes in terms of the most probable input symbol, where compression efficiency is determined by the nonlinear codeword length objective of…
In this work, we study the problem of finding the asymptotic growth rate of the number of of $d$-dimensional arrays with side length $n$ over a given alphabet which avoid a list of one-dimensional "forbidden" words along all cardinal…
This paper presents a general framework for exploiting the representational capacity of neural networks to approximate complex, nonlinear reward functions in the context of solving the inverse reinforcement learning (IRL) problem. We show…
The weak law of large numbers implies that, under mild assumptions on the source, the Renyi entropy per produced symbol converges (in probability) towards the Shannon entropy rate. This paper quantifies the speed of this convergence for…
Rich words are characterized by containing the maximum possible number of distinct palindromes. Several characteristic properties of rich words have been studied; yet the analysis of repetitions in rich words still involves some interesting…
The Matrix-based Renyi's entropy enables us to directly measure information quantities from given data without the costly probability density estimation of underlying distributions, thus has been widely adopted in numerous statistical…
The subword complexity of a finite word $w$ of length $N$ is a function which associates to each $n\le N$ the number of all distinct subwords of $w$ having the length $n$. We define the \emph{maximal complexity} C(w) as the maximum of the…
We prove that a uniformly random automaton with $n$ states on a 2-letter alphabet has a synchronizing word of length $O(n^{1/2}\log n)$ with high probability (w.h.p.). That is to say, w.h.p. there exists a word $\omega$ of such length, and…
We show that the number of all maximal $\alpha$-gapped repeats and palindromes of a word of length $n$ is at most $3(\pi^2/6 + 5/2) \alpha n$ and $7 (\pi^2 / 6 + 1/2) \alpha n - 5 n - 1$, respectively.
We investigate the performance of large language models on repetitive deterministic prediction tasks and study how the sequence accuracy rate scales with output length. Each such task involves repeating the same operation n times. Examples…
When deploying artificial agents in real-world environments where they interact with humans, it is crucial that their behavior is aligned with the values, social norms or other requirements of that environment. However, many environments…
In 2017, Vesti proposed the problem of determining the repetition threshold for infinite rich words, i.e., for infinite words in which all factors of length $n$ contain $n$ distinct nonempty palindromic factors. In 2020, Currie, Mol, and…