Related papers: A remark on conditional entropy

With the attention mechanism, transformers achieve significant empirical successes. Despite the intuitive understanding that transformers perform relational inference over long sequences to produce desirable representations, we lack a…

Entropic causal inference is a recent framework for learning the causal graph between two variables from observational data by finding the information-theoretically simplest structural explanation of the data, i.e., the model with smallest…

In this Letter we show that the time reversal asymmetry of a stationary time series provides information about the entropy production of the physical mechanism generating the series, even if one ignores any detail of that mechanism. We…

All of the basic microsopic physical laws are time reversible. In contrast, the second law of thermodynamics, which is a macroscopic physical representation of the world, is able to describe irreversible processes in an isolated system…

We consider the structural change in a class of discrete valued time series that the conditional distribution follows a one-parameter exponential family. We propose a change-point test based on the maximum likelihood estimator of the…

We show for a large class of interacting particle systems that whenever the stationary measure is not reversible for the dynamics, then the mean entropy production in the steady state is strictly positive. This extends to the thermodynamic…

Quantum superposition, a cornerstone of quantum mechanics, enables systems to exist in multiple states simultaneously, giving rise to probabilistic outcomes. In quantum information science, conditional entropy has become a key metric for…

Entropic dynamics is a framework for defining dynamical systems that is aligned with the principles of information theory. In an entropic dynamics model for motion on a statistical manifold, we find that the rate of changes for expected…

When evaluating causal influence from one time series to another in a multivariate dataset it is necessary to take into account the conditioning effect of the other variables. In the presence of many variables, and possibly of a reduced…

A method for analyzing sequential data sets, similar to the permutation entropy one, is discussed. The characteristic features of this method are as follows: it preserves information about equal values, if any, in the embedding vectors; it…

The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible while constrained to match empirically estimated feature expectations. However, in many real-world…

The conditional distribution of the next outcome given the infinite past of a stationary process can be inferred from finite but growing segments of the past. Several schemes are known for constructing pointwise consistent estimates, but…

Classifier-guided diffusion models generate conditional samples by augmenting the reverse-time score with the gradient of the log-probability predicted by a probabilistic classifier. In practice, this classifier is usually obtained by…

We study the temporal approach to equilibrium of the Gibbs' and conditional entropies for both invertible deterministic dynamics as well as non-invertible stochastic systems in the presence of white noise. The conditional entropy will…

In the dynamics of open quantum systems, the backflow of information to the reduced system under study has been suggested as the actual physical mechanism inducing memory and thus leading to non-Markovian quantum dynamics. To this aim, the…

Previously proposed measures of entanglement, such as entanglement of formation and assistance, are shown to be special cases of the relative entropy of entanglement. The difference between these measures for an ensemble of mixed states is…

Entropy and differential entropy are important quantities in information theory. A tractable extension to singular random variables-which are neither discrete nor continuous-has not been available so far. Here, we present such an extension…

We apply a common measure of randomness, the entropy, in the context of iterated functions on a finite set with n elements. For a permutation, it turns out that this entropy is asymptotically (for a growing number of iterations) close to…

Permutation entropy quantifies the diversity of possible orderings of the values a random or deterministic system can take, as Shannon entropy quantifies the diversity of values. We show that the metric and permutation entropy…

Conditional mutual information is important in the selection and interpretation of graphical models. Its empirical version is well known as a generalised likelihood ratio test and that it may be represented as a difference in entropy. We…