Related papers: Composition Law of Conjugate Observables in Random…
We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our…
Scaling laws aim to accurately predict model performance across different scales. Existing scaling-law studies almost exclusively rely on cross-entropy as the evaluation metric. However, cross-entropy provides only a partial view of…
We propose a framework for joint entropy coding and encryption using Chaotic maps. We begin by observing that the message symbols can be treated as the symbolic sequence of a discrete dynamical system. For an appropriate choice of the…
We consider the problem of distilling uniform random bits from an unknown source with a given $p$-entropy using linear hashing. As our main result, we estimate the expected $p$-divergence from the uniform distribution over the ensemble of…
We prove the full equivalence between Assembly Theory (AT) and Shannon Entropy via a method based upon the principles of statistical compression renamed `assembly index' that belongs to the LZ family of popular compression algorithms (ZIP,…
Preserving biodiversity and ecosystem stability is a challenge that can be pursued through modern statistical mechanics modeling. Here we introduce a variational maximum entropy-based algorithm to evaluate the entropy in a minimal ecosystem…
We study universal traits which emerge both in real-world complex datasets, as well as in artificially generated ones. Our approach is to analogize data to a physical system and employ tools from statistical physics and Random Matrix Theory…
We introduce a novel entropy-related function, \textit{non-repeatability}, designed to capture dynamical behaviors in complex systems. Its normalized form, \textit{mutability}, has been previously applied in statistical physics as a…
Sampling permutations from S_n is a fundamental problem from probability theory. The nearest neighbor transposition chain \cal{M}}_{nn} is known to converge in time \Theta(n^3 \log n) in the uniform case and time \Theta(n^2) in the constant…
The NP-complete Permutation Pattern Matching problem asks whether a permutation P (the pattern) can be matched into a permutation T (the text). A matching is an order-preserving embedding of P into T. In the Generalized Permutation Pattern…
In this paper we consider the relation between random sums and compositions of different processes. In particular, for independent Poisson processes $N_\alpha(t)$, $N_\beta(t)$, $t>0$, we show that $N_\alpha(N_\beta(t))…
The paper characterizes the Shannon (1948) and Tsallis (1988) entropies in a standard framework of decision theory, mixture sets. Procedural mixture sets are introduced as a variant of mixture sets in which it is not necessarily true that a…
On a finite outcome space, the canonical Gibbs distribution is usually obtained by maximizing Shannon entropy at fixed mean of an externally supplied energy functional. This paper studies the finite-state consequences of a ratio-cost…
We comment on a formulation of quantum statistical mechanics, which incorporates the statistical inference of Shannon. Our basic idea is to distinguish the dynamical entropy of von Neumann, $H = -k Tr \hat{\rho}\ln\hat{\rho}$, in terms of…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
In this article, we show that a linear combination $X$ of $n$ independent, unbiased Bernoulli random variables $\{X_k\}$ can match the first $2n$ moments of a random variable $Y$ which is uniform on an interval. More generally, for each $p…
Compressed Counting (CC), based on maximally skewed stable random projections, was recently proposed for estimating the p-th frequency moments of data streams. The case p->1 is extremely useful for estimating Shannon entropy of data…
The ability to continually learn, retain and deploy skills to accomplish goals is a key feature of intelligent and efficient behavior. However, the neural mechanisms facilitating the continual learning and flexible (re-)composition of…
Counting permutations of $[n]$ by the number of records, i.e. left-to-right maxima, is a classic problem in combinatorial enumeration. In the first volume of ``The Art of Computer Programming", Donald Knuth demonstrated its relevance for…
Observational entropy -- a quantity that unifies Boltzmann's entropy, Gibbs' entropy, von Neumann's macroscopic entropy, and the diagonal entropy -- has recently been argued to play a key role in a modern formulation of statistical…