Related papers: Entropy and Long range correlations in literary En…
Selecting an appropriate book is crucial for fostering reading habits in children. While children exhibit varying levels of complexity when generating oral narratives, the question arises: do children's books also differ in narrative…
Entanglement entropies have revealed, in the last years, to be a powerful tool to extract information about the physics of condensed-matter systems. In the first part of this thesis, we show how to extract essential details about the…
We develop information-theoretic measures of spatial structure and pattern in more than one dimension. As is well known, the entropy density of a two-dimensional configuration can be efficiently and accurately estimated via a converging…
We examine a class of deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical…
We study the correlations of classical and quantum systems from the information theoretical points of view. We analyze a simple measure of correlations based on entropy (such measure was already investigated as the degree of entanglement by…
We bound the change in entropy incurred by an irreducible subshift of finite type upon perturbing it by forbidding a pair of admissible words. Lind has proven such bounds in the one-word case, and we adapt his methods. In particular, we…
Let $X=(X_i)_{i\ge 1}$ and $Y=(Y_i)_{i\ge 1}$ be two sequences of independent and identically distributed (iid) random variables taking their values, uniformly, in a common totally ordered finite alphabet. Let LCI$_n$ be the length of the…
We study entanglement entropy (EE) for a Maxwell field in 2+1 dimensions. We do numerical calculations in two dimensional lattices. This gives a concrete example of the general results of our recent work on entropy for lattice gauge fields…
Training long-context language models to capture long-range dependencies requires specialized data construction. Current approaches, such as generic text concatenation or heuristic-based variants, frequently fail to guarantee genuine…
New families of Fisher information and entropy power inequalities for sums of independent random variables are presented. These inequalities relate the information in the sum of $n$ independent random variables to the information contained…
The distribution of word probabilities in the monkey model of Zipf's law is associated with two universality properties: (1) the power law exponent converges strongly to $-1$ as the alphabet size increases and the letter probabilities are…
In this paper, we present a novel method for computing the relative entropy as well as the expected relative entropy using an MCMC chain. The relative entropy from information theory can be used to quantify differences in posterior…
Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a…
We study a deliberately simple, fully non-linguistic model of text: a sequence of independent draws from a finite alphabet of letters plus a single space symbol. A word is defined as a maximal block of non-space symbols. Within this…
Information capacity of a symbol sequence is a measure of the unexpectedness of a continuation of given string of symbols. Continuation of a string is determined through the maximum entropy of the reconstructed frequency dictionary; the…
The complexity function of an infinite word $w$ on a finite alphabet $A$ is the sequence counting, for each non-negative $n$, the number of words of length $n$ on the alphabet $A$ that are factors of the infinite word $w$. For any given…
Of basic interest is the quantification of the long term growth of a language's lexicon as it develops to more completely cover both a culture's communication requirements and knowledge space. Here, we explore the usage dynamics of words in…
Entropy and differential entropy are important quantities in information theory. A tractable extension to singular random variables-which are neither discrete nor continuous-has not been available so far. Here, we present such an extension…
Understanding how ideas relate to each other is a fundamental question in many domains, ranging from intellectual history to public communication. Because ideas are naturally embedded in texts, we propose the first framework to…
When an i.i.d.\ sequence of letters is cut into words according to i.i.d.\ renewal times, an i.i.d.\ sequence of words is obtained. In the \emph{annealed} LDP (large deviation principle) for the empirical process of words, the rate function…