Related papers: Scaling behaviour of entropy estimates
Random sequences attain the highest entropy rate. The estimation of entropy rate for an ergodic source can be done using the Lempel Ziv complexity measure yet, the exact entropy rate value is only reached in the infinite limit. We prove…
We inspect the deductive connection between the neural scaling law and Zipf's law -- two statements discussed in machine learning and quantitative linguistics. The neural scaling law describes how the cross entropy rate of a foundation…
We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our…
It is well known that, for chaotic systems, the production of relevant entropy (Boltzmann-Gibbs) is always linear and the system has strong (exponential) sensitivity to initial conditions. In recent years, various numerical results indicate…
We derive upper and lower bounds on the overall compression ratio of the 1978 Lempel-Ziv (LZ78) algorithm, applied independently to $k$-blocks of a finite individual sequence. Both bounds are given in terms of normalized empirical entropies…
We address this work to investigate symbolic sequences with long-range correlations by using computational simulation. We analyze sequences with two, three and four symbols that could be repeated $l$ times, with the probability distribution…
The rank-size regularity known as Zipf's law is one of scaling laws and frequently observed within the natural living world and in social institutions. Many scientists tried to derive the rank-size scaling relation by entropy-maximizing…
Scaling laws aim to accurately predict model performance across different scales. Existing scaling-law studies almost exclusively rely on cross-entropy as the evaluation metric. However, cross-entropy provides only a partial view of…
The goal of this paper is to develop an estimate for the entropy of random long-range correlated symbolic sequences with elements belonging to a finite alphabet. As a plausible model, we use the high-order additive stationary ergodic Markov…
Exactly solving a spinless fermionic system in two and three dimensions, we investigate the scaling behavior of the block entropy in critical and non-critical phases. The scaling of the block entropy crucially depends on the nature of the…
We investigate symbolic sequences and in particular information carriers as e.g. books and DNA--strings. First the higher order Shannon entropies are calculated, a characteristic root law is detected. Then the algorithmic entropy is…
We focus on the FeigenbaumCoulletTresser point of the dissipative one-dimensional z logistic map. We show that sums of iterates converge to q Gaussian distributions, which optimize the nonadditive entropic functional Sq under simple…
In this overview, we discuss the connections between the observations of critical dynamics in neuronal networks and the maximum entropy models that are often used as statistical models of neural activity, focusing in particular on the…
In this work, building on state-of-the-art quantum Monte Carlo simulations, we perform systematic finite-size scaling of both entanglement and participation entropies for long-range Heisenberg chain with unfrustrated power-law decaying…
We investigate symbolic sequences and in particular information carriers as e.g. books and DNA-strings. First the higher order Shannon entropies are calculated, a characteristic root law is detected. Then the algorithmic entropy is…
Ensemble of initial conditions for nonlinear maps can be described in terms of entropy. This ensemble entropy shows an asymptotic linear growth with rate K. The rate K matches the logarithm of the corresponding asymptotic sensitivity to…
The Lempel-Ziv universal coding scheme is asymptotically optimal for the class of all stationary ergodic sources. A problem of robustness of this property under small violations of ergodicity is studied. A notion of deficiency of…
A recently introduced model of coupled non linear oscillators in a ring is revisited in terms of its information processing capabilities. The use of Lempel-Ziv based entropic measures allows to study thoroughly the complex patterns…
In this paper, we discuss the possibility of unexplored behaviours for the entanglement entropy in extended quantum systems. Namely, we study the R\'enyi entanglement entropy for the ground state of long-range Kitaev chains with slow…
We apply renormalized entropy as a complexity measure to the logistic and sine-circle maps. In the case of logistic map, renormalized entropy decreases (increases) until the accumulation point (after the accumulation point up to the most…