Related papers: Large Alphabets and Incompressibility
We investigated long range correlations in two literary texts, Moby Dick by H. Melville and Grimm's tales. The analysis is based on the calculation of entropy like quantities as the mutual information for pairs of letters and the entropy,…
We consider an \eps-approximation by n-term partial sums of the Karhunen-Lo\`eve expansion to d-parametric random fields of tensor product-type in the average case setting. We investigate the behavior, as d tends to infinity, of the…
We define an algorithm that parses multidimensional arrays sequentially into mainly unrepeated but nested multidimensional sub-arrays of increasing size, and show that the resulting sub-block pointer encoder compresses almost every…
A new framework is introduced for examining and evaluating the fundamental limits of lossless data compression, that emphasizes genuinely non-asymptotic results. The {\em sample complexity} of compressing a given source is defined as the…
Evaluating whether large language models (LLMs) capture the structure of natural language beyond local fluency remains an open challenge. Existing evaluation methods, largely based on task performance or short-context behavior, provide…
Systematic generalization remains challenging for current language models, which are known to be both sensitive to semantically similar permutations of the input and to struggle with known concepts presented in novel contexts. Although…
Large language models (LLMs) are widely recognized for their exceptional capacity to capture semantics meaning. Yet, there remains no established metric to quantify this capability. In this work, we introduce a quantitative metric,…
This paper studies the complexity of estimating Renyi divergences of discrete distributions: $p$ observed from samples and the baseline distribution $q$ known \emph{a priori}. Extending the results of Acharya et al. (SODA'15) on estimating…
In natural language processing, the entropy of a language is a measure of its unpredictability and complexity. The first study on this subject was conducted by Claude Shannon in 1951. By having participants predict the next character in a…
This work is concerned with regular languages defined over large alphabets, either infinite or just too large to be expressed enumeratively. We define a generic model where transitions are labeled by elements of a finite partition of the…
We describe general approach to classification of character sequences (texts, DNA) using relative entropy estimated by off-the-shelf compression and Markov Chains and find them precise enough. We also notice that the method for estimating…
We review the recent progress in the investigation of powerfree words, with particular emphasis on binary cubefree and ternary squarefree words. Besides various bounds on the entropy, we provide bounds on letter frequencies and consider…
Given a reference computer, Kolmogorov complexity is a well defined function on all binary strings. In the standard approach, however, only the asymptotic properties of such functions are considered because they do not depend on the…
We study the problem of exact sampling under an exponential communication cost, specifically Campbell's average codeword length $L(t)$ of order $t$, and R\'enyi's entropy. We provide a lower bound on the Campbell cost of exact sampling that…
We review several statistical complexity measures proposed over the last decade and a half as general indicators of structure or correlation. Recently, Lopez-Ruiz, Mancini, and Calbet [Phys. Lett. A 209 (1995) 321] introduced another…
We generalise a multiple string pattern matching algorithm, recently proposed by Fredriksson and Grabowski [J. Discr. Alg. 7, 2009], to deal with arbitrary dictionaries on an alphabet of size $s$. If $r_m$ is the number of words of length…
Permutation entropy quantifies the diversity of possible orderings of the values a random or deterministic system can take, as Shannon entropy quantifies the diversity of values. We show that the metric and permutation entropy…
The paper makes the observation that all orders of information entropy are equal in signals composed of repeating units of distinct symbols where the units can be classified as a member of a symmetry group. This leads to an improved metric…
Kahn and Kim (J. Comput. Sci., 1995) have shown that for a finite poset $P$, the entropy of the incomparability graph of $P$ (normalized by multiplying by the order of $P$) and the base-$2$ logarithm of the number of linear extensions of…
Large language models (LLMs) are increasingly deployed on complex reasoning tasks, yet little is known about their ability to internally evaluate problem difficulty, which is an essential capability for adaptive reasoning and efficient…