相关论文: Large Alphabets and Incompressibility

We investigate symbolic sequences and in particular information carriers as e.g. books and DNA-strings. First the higher order Shannon entropies are calculated, a characteristic root law is detected. Then the algorithmic entropy is…

We conceptualize the process of understanding as information compression, and propose a method for ranking large language models (LLMs) based on lossless data compression. We demonstrate the equivalence of compression length under…

In this note, we show that the relative entropy of an empirical distribution of $n$ samples drawn from a set of size $k$ with respect to the true underlying distribution is exponentially concentrated around its expectation, with central…

Consider a system of $n$ weakly interacting particles driven by independent Brownian motions. In many instances, it is well known that the empirical measure converges to the solution of a partial differential equation, usually called…

We study the compressibility of enumerations in the context of Kolmogorov complexity, focusing on strong and weak forms of compression and their gain: the amount of auxiliary information embedded in the compressed enumeration. The existence…

An algebraic integer is said large if all its real or complex embeddings have absolute value larger than $1$. An integral ideal is said \emph{large} if it admits a large generator. We investigate the notion of largeness, relating it to some…

For information-theoretic quantities with an asymptotic operational characterization, the question arises whether an alternative single-shot characterization exists, possibly including an optimization over an ancilla system. If the…

Relative entropy is a fundamental class of distances between probability distributions, with widespread applications in probability theory, statistics, and machine learning. In this work, we study relative entropy from a categorical…

The concept of effective complexity of an object as the minimal description length of its regularities has been initiated by Gell-Mann and Lloyd. The regularities are modeled by means of ensembles, that is probability distributions on…

As it is known, universal codes, which estimate the entropy rate consistently, exist for stationary ergodic sources over finite alphabets but not over countably infinite ones. We generalize universal coding as the problem of universal…

We give new general formulas for the asymptotics of the number of spanning trees of a large graph. A special case answers a question of McKay (1983) for regular graphs. The general answer involves a quantity for infinite graphs that we call…

We consider in this paper the problem of computing the entropy of a braid. We recall its definition and construct, for each braid, a sequence of real numbers, whose limit is its entropy. We state one conjecture about the convergence speed,…

A recent result presented the expansion for the entropy rate of a Hidden Markov Process (HMP) as a power series in the noise variable $\eps$. The coefficients of the expansion around the noiseless ($\eps = 0$) limit were calculated up to…

A well-known fact in the field of lossless text compression is that high-order entropy is a weak model when the input contains long repetitions. Motivated by this, decades of research have generated myriads of so-called dictionary…

In [3] a short proof is given that some strings have maximal plain Kolmogorov complexity but not maximal prefix-free complexity. The proof uses Levin's symmetry of information, Levin's formula relating plain and prefix complexity and Gacs'…

In the Shortest Superstring problem we are given a set of strings $S=\{s_1, \ldots, s_n\}$ and integer $\ell$ and the question is to decide whether there is a superstring $s$ of length at most $\ell$ containing all strings of $S$ as…

We consider the problem of constructing prefix-free codes in which a designated symbol, a space, can only appear at the end of codewords. We provide a linear-time algorithm to construct almost-optimal codes with this property, meaning that…

There is a parallelism between Shannon information theory and algorithmic information theory. In particular, the same linear inequalities are true for Shannon entropies of tuples of random variables and Kolmogorov complexities of tuples of…

As is the case of many signals produced by complex systems, language presents a statistical structure that is balanced between order and disorder. Here we review and extend recent results from quantitative characterisations of the degree of…

Large language models (LLMs) have shown significant progress in reasoning tasks. However, recent studies show that transformers and LLMs fail catastrophically once reasoning problems exceed modest complexity. We revisit these findings…