Related papers: The Entropy of Parallel Systems
There are three ways to conceptualize entropy: entropy as an extensive thermodynamic quantity of physical systems (Clausius, Boltzmann, Gibbs), entropy as a measure for information production of ergodic sources (Shannon), and entropy as a…
A new approach to estimate the Shannon entropy of a long-range correlated sequence is proposed. The entropy is written as the sum of two terms corresponding respectively to power-law (\emph{ordered}) and exponentially (\emph{disordered})…
This chapter provides a comprehensive and self-contained discussion of the most recent developments of information theory of networks. Maximum entropy models of networks are the least biased ensembles enforcing a set of constraints and are…
R\'esum\'e: Le principal objet de cette communication est de faire une r\'etro perspective succincte de l'utilisation de l'entropie et du principe du maximum d'entropie dans le domaine du traitement du signal. Apr\`es un bref rappel de…
Entropy is critically examined as a fundamental concept in contemporary science and informatics. Although the typical Shannon entropy provides a proper framework for describing the canonical ensemble, it fails to represent adequately the…
We apply renormalized entropy as a complexity measure to the logistic and sine-circle maps. In the case of logistic map, renormalized entropy decreases (increases) until the accumulation point (after the accumulation point up to the most…
Systematic generalization remains challenging for current language models, which are known to be both sensitive to semantically similar permutations of the input and to struggle with known concepts presented in novel contexts. Although…
Entropy is a central concept in physics, but can be challenging to calculate even for systems that are easily simulated. This is exacerbated out of equilibrium, where generally little is known about the distribution characterizing simulated…
The Shannon entropy is a widely used summary statistic, for example, network traffic measurement, anomaly detection, neural computations, spike trains, etc. This study focuses on estimating Shannon entropy of data streams. It is known that…
This paper describes an entropy equation, but one that should be used for measuring energy and not information. In relation to the human brain therefore, both of these quantities can be used to represent the stored information. The human…
In this paper, we demonstrate how the physics of entropy production, when combined with symmetry constraints, can be used for implementing high-performance and energy-efficient analog computing systems. At the core of the proposed framework…
For a closed-loop control system with a digital channel between the sensor and the controller, the notion of invariance entropy quantifies the smallest average rate of information transmission above which a given compact subset of the state…
Entanglement, which quantifies non-local correlations in quantum mechanics, is the fascinating concept behind much of aspiration towards quantum technologies. Nevertheless, directly measuring the entanglement of a many-particle system is…
The analysis of execution paths (also known as software traces) collected from a given software product can help in a number of areas including software testing, software maintenance and program comprehension. The lack of a scalable…
Nonequilibrium complex systems are often effectively described by the mixture of different dynamics on different time scales. Superstatistics, which is "statistics of statistics" with two largely separated time scales, offers a consistent…
The weak law of large numbers implies that, under mild assumptions on the source, the Renyi entropy per produced symbol converges (in probability) towards the Shannon entropy rate. This paper quantifies the speed of this convergence for…
The structure of the network can be described by motifs, which are subgraphs that often repeat themselves. In order to understand the structure of network motifs, it is of great importance to study subgraphs from the perspective of…
Uncertainty principle plays a crucial role in quantum mechanics, because it captures the essence of the inevitable randomness associated with the outcomes of two incompatible quantum measurements. Information entropy can perfectly describe…
In this paper, I expand Shannon's definition of entropy into a new form of entropy that allows integration of information from different random events. Shannon's notion of entropy is a special case of my more general definition of entropy.…
The uncertainty principle is an inherent characteristic of quantum mechanics. This principle can be formulated in various form. Fundamentally, this principle can be expressed in terms of the standard deviation of the measured observables.…