Related papers: The LZ78 Source
There are three ways to conceptualize entropy: entropy as an extensive thermodynamic quantity of physical systems (Clausius, Boltzmann, Gibbs), entropy as a measure for information production of ergodic sources (Shannon), and entropy as a…
We lift important results about universally typical sets, typically sampled sets, and empirical entropy estimation in the theory of samplings of discrete ergodic information sources from the usual one-dimensional discrete-time setting to a…
We consider a generic class of stochastic particle-based models whose state at an instant in time is described by a set of continuous degrees of freedom (e.g. positions), and the length of this set changes stochastically in time due to…
This paper studies the Shannon regime for the random displacement of stationary point processes. Let each point of some initial stationary point process in $\R^n$ give rise to one daughter point, the location of which is obtained by adding…
Even simply-defined, finite-state generators produce stochastic processes that require tracking an uncountable infinity of probabilistic features for optimal prediction. For processes generated by hidden Markov chains the consequences are…
Autoregressive generative models -- including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba -- all generate sequences by sampling each output from a deterministic summary of the past,…
Transfer entropy, an information-theoretic measure of time-directed information transfer between joint processes, has steadily gained popularity in the analysis of complex stochastic dynamics in diverse fields, including the neurosciences,…
In a previous paper, we introduced an axiomatic system for information thermodynamics, deriving an entropy function that includes both thermodynamic and information components. From this function we derived an entropic probability…
This work provides new results for the analysis of random sequences in terms of $\ell_p$-compressibility. The results characterize the degree in which a random sequence can be approximated by its best $k$-sparse version under different…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
We illustrate the mathematical theory of entropy production in repeated quantum measurement processes developed in a previous work by studying examples of quantum instruments displaying various interesting phenomena and singularities. We…
This study explores a Gaussian quasi-likelihood approach for estimating parameters of diffusion processes with Markovian regime switching. Assuming the ergodicity under high-frequency sampling, we will show the asymptotic normality of the…
The entropy of an ergodic finite-alphabet process can be computed from a single typical sample path x_1^n using the entropy of the k-block empirical probability and letting k grow with $n$ roughly like log n. We further assume that the…
We construct intrinsic on-and off-diagonal upper and lower estimates for the transition probability density of a L\'evy process in small time. By intrinsic we mean that such estimates reflect the structure of the characteristic exponent of…
Let $(\Omega, \mathcal{F}, (\mathcal{F})_{t\ge 0}, P)$ be a complete stochastic basis, $X$ a semimartingale with predictable compensator $(B, C, \nu)$. Consider a family of probability measures $\mathbf{P}=( {P}^{n, \psi}, \psi\in \Psi,…
The standard Large Deviation Theory (LDT) mirrors the Boltzmann-Gibbs (BG) factor which describes the thermal equilibrium of short-range Hamiltonian systems, the velocity distribution of which is Maxwellian. It is generically applicable to…
Using the diagonal entropy, we analyze the dynamical signatures of the Lipkin-Meshkov-Glick (LMG) model excited-state quantum phase transition (ESQPT). We first show that the time evolution of the diagonal entropy behaves as an efficient…
We construct multiperiodic processes -- a simple example of stationary ergodic (but not mixing) processes over natural numbers that enjoy the vanishing entropy rate under a mild condition. Multiperiodic processes are supported on randomly…
We study the entropy production of a system with a finite number of states connected by random transition rates. The stationary entropy production, driven out of equilibrium both by asymmetric transition rates and by an external probability…
Let $(X_{\underline{\ell}})_{\underline{\ell} \in \mathbb Z^d}$ be a real random field (r.f.) indexed by $\mathbb Z^d$ with common probability distribution function $F$. Let $(z_k)_{k=0}^\infty$ be a sequence in $\mathbb Z^d$. The empirical…