Related papers: Entropy and long-range correlations in random symb…
In this paper, we extend a symbolic association framework for being able to handle missing elements in multimodal sequences. The general scope of the work is the symbolic associations of object-word mappings as it happens in language…
The aim of this paper is to study the correlation properties of random sequences with additive linear conditional probability distribution function (CPDF) and elaborate a reliable tool for their generation. It is supposed that the state…
We study the entropy production rate in systems described by linear Langevin equations, containing mixed even and odd variables under time reversal. Exact formulas are derived for several important quantities in terms only of the means and…
The paper describes an approach to measuring convergence of an algorithm to its result in terms of an entropy-like function of partitions of its inputs of a given length. The goal is to look at the algorithmic data processing from the…
Local correlation entropy, introduced by Takens in 1983, represents the exponential decay rate of the relative frequency of recurrences in the trajectory of a point, as the embedding dimension grows to infinity. In this paper we study…
This study re-evaluates the assumption that long-range correlations in sentence length are a fundamental feature of natural language and a marker of literary style. While previous research has suggested that punctuation marks--particularly…
We introduce a family of scale-invariant entropy statistics derived from logarithmically aggregated distance distributions of point processes, with prime numbers serving as a motivating example. The construction associates to each finite…
We propose a formal expansion of the transfer entropy to put in evidence irreducible sets of variables which provide information for the future state of each assigned target. Multiplets characterized by a large contribution to the expansion…
The growing study of time series, especially those related to nonlinear systems, has challenged the methodologies to characterize and classify dynamical structures of a signal. Here we conceive a new diagnostic tool for time series based on…
Entropic measures provide analytic tools to help us understand correlation in quantum systems. In our previous work, we calculated linear entropy and von Neumann entropy as entanglement measures for the ground state and lower lying excited…
The entanglement entropy in clean, as well as in random quantum spin chains has a logarithmic size-dependence at the critical point. Here, we study the entanglement of composite systems that consist of a clean and a random part, both being…
Entropy production is a universal measure of irreversibility and energy dissipation in physical, chemical, and biological systems operating far from equilibrium. However, quantifying and spatiotemporally localising it in complex processes…
In large language models (LLMs), each block operates on the residual stream to map input token sequences to output token distributions. However, most of the interpretability literature focuses on internal latent representations, leaving…
We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in…
Inferring models, predicting the future, and estimating the entropy rate of discrete-time, discrete-event processes is well-worn ground. However, a much broader class of discrete-event processes operates in continuous-time. Here, we provide…
Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream tasks. However, applying BPE to unsegmented languages such…
We define a new quantitative measure for an arbitrary factorial language: the entropy of a random walk in the prefix tree associated with the language; we call it Markov entropy. We relate Markov entropy to the growth rate of the language…
A powerful tool is developed for the characterization of chaotic signals. The approach is based on the symbolic encoding of time series (according to their ordinal patterns) combined with the ensuing characterization of the corresponding…
We apply a common measure of randomness, the entropy, in the context of iterated functions on a finite set with n elements. For a permutation, it turns out that this entropy is asymptotically (for a growing number of iterations) close to…
An effective way to scale up test-time compute of large language models is to sample multiple responses and then select the best one, as in Grok Heavy and Gemini Deep Think. Existing selection methods often rely on external reward models,…