Related papers: Relative Entropy, Probabilistic Inference and AI
Bayes' theorem incorporates distinct types of information through the likelihood and prior. Direct observations of state variables enter the likelihood and modify posterior probabilities through consistent updating. Information in terms of…
Many AI researchers argue that probability theory is only capable of dealing with uncertainty in situations where a full specification of a joint probability distribution is available, and conclude that it is not suitable for application in…
An observer increases in relative entropy as it receives information from what it is observing. In a system of only an observer and the observed, an increase in the relative entropy of the observer is a decrease in the relative entropy of…
In this article we describe the applications of the relative entropy framework. In particular uniqueness of an entropy solution is proven for a scalar conservation law, using the notion of measure-valued entropy solutions. Further we survey…
We propose an intersubjective epistemic approach to foundations of probability theory and statistical inference, based on relative entropy and category theory, and aimed to bypass the mathematical and conceptual problems of existing…
Information-theoretic (IT) measures are ubiquitous in artificial intelligence: entropy drives decision-tree splits and uncertainty quantification, cross-entropy is the default classification loss, mutual information underpins representation…
We introduce the relative tail entropy to establish a variational principle for continuous bundle random dynamical systems. We also show that the relative tail entropy is conserved by the principal extension.
The present paper studies continuity of generalized entropy functions and relative entropies defined using the notion of a deformed logarithmic function. In particular, two distinct definitions of relative entropy are discussed. As an…
The paper presents a lightweight discussion of relations between energy, entropy, information, and intelligence, based on an analysis of the energy needed for computation.
Model or variable selection is usually achieved through ranking models according to the increasing order of preference. One of methods is applying Kullback-Leibler distance or relative entropy as a selection criterion. Yet that will raise…
Entropy is a central concept in physics, but can be challenging to calculate even for systems that are easily simulated. This is exacerbated out of equilibrium, where generally little is known about the distribution characterizing simulated…
Transfer entropy is a widely used measure for quantifying directed information flows in complex systems. While the challenges of estimating transfer entropy for continuous data are well known, it has two major shortcomings for data of…
The following three sections and appendices are taken from my thesis "The Foundations of Inference and its Application to Fundamental Physics" from 2021, in which I construct a theory of entropic inference from first principles. The…
The method of optimizing entropy is used to (i) conduct Asymptotic Hypothesis Testing and (ii) determine the particle distribution for which Entropy is maximized. This paper focuses on two related applications of Information Theory:…
We introduce the notion of relative volume entropy for two spacetimes with preferred compact spacelike foliations. This is accomplished by applying the notion of Kullback-Leibler divergence to the volume elements induced on spacelike…
Entropy notions for $\varepsilon$-incremental practical stability and incremental stability of deterministic nonlinear systems under disturbances are introduced. The entropy notions are constructed via a set of points in state space which…
Statistical mechanics relies on the complete though probabilistic description of a system in terms of all the microscopic variables. Its object is to derive therefrom static and dynamic properties involving some reduced set of variables.…
This paper represents an extended version of an earlier note [10]. The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. We analyse analogs of…
This report introduces general ideas and some basic methods of the Bayesian probability theory applied to physics measurements. Our aim is to make the reader familiar, through examples rather than rigorous formalism, with concepts such as:…
We show that Skilling's method of induction leads to a unique general theory of inductive inference, the method of Maximum relative Entropy (ME). The main tool for updating probabilities is the logarithmic relative entropy; other entropies…