Related papers: Source separation as an exercise in logical induct…
Bayes' theorem incorporates distinct types of information through the likelihood and prior. Direct observations of state variables enter the likelihood and modify posterior probabilities through consistent updating. Information in terms of…
Today, the vast majority of the world's digital information is represented using the fundamental assumption, introduced by Claude Shannon in 1948, that ``...the semantic aspects of communication are irrelevant to the engineering problem (of…
The significance of statistical physics concepts such as entropy extends far beyond classical thermodynamics. We interpret the similarity between partitions in statistical mechanics and partitions in Bayesian inference as an articulation of…
The problem of induction has persisted since Hume exposed the logical gap between repeated observation and universal inference. Traditional attempts to resolve it have oscillated between two extremes: the probabilistic optimism of Laplace…
Information based thermodynamic logic is revisited. It consists of two parts: Part A applies the modern theory of probability in which an arbitrary convex function \phi is employed as an analytic "device" to express information as…
This work develops problem statements related to encoders and autoencoders with the goal of elucidating variational formulations and establishing clear connections to information-theoretic concepts. Specifically, four problems with varying…
Accurately determining dependency structure is critical to discovering a system's causal organization. We recently showed that the transfer entropy fails in a key aspect of this---measuring information flow---due to its conflation of dyadic…
Clarke and Barron analysed the relative entropy between an i.i.d. source and a Bayesian mixture over a continuous class containing that source. In this paper a comparable result is obtained when the source is permitted to be both…
Information theory is concerned with the study of transmission, processing, extraction, and utilization of information. In its most abstract form, information is conceived as a means of resolving uncertainty. Shannon and Weaver (1949) were…
We describe and develop a close relationship between two problems that have customarily been regarded as distinct: that of maximizing entropy, and that of minimizing worst-case expected loss. Using a formulation grounded in the equilibrium…
This paper extends the work of Clarke [1] on the Bayesian foundations of the biomagnetic inverse problem. It derives expressions for the expectation and variance of the a posteriori source current probability distribution given a prior…
Separation logic is a substructural logic which has proved to have numerous and fruitful applications to the verification of programs working on dynamic data structures. Recently, Barthe, Hsu and Liao have proposed a new way of giving…
I discuss the design of the method of entropic inference as a general framework for reasoning under conditions of uncertainty. The main contribution of this discussion is to emphasize the pragmatic elements in the derivation. More…
Experiments in research on memory, language, and in other areas of cognitive science are increasingly being analyzed using Bayesian methods. This has been facilitated by the development of probabilistic programming languages such as Stan,…
This chapter provides a overview of Bayesian inference, mostly emphasising that it is a universal method for summarising uncertainty and making estimates and predictions using probability statements conditional on observed data and an…
The ``Gibbs Paradox'' refers to several related questions concerning entropy in thermodynamics and statistical mechanics: whether it is an extensive quantity or not, how it changes when identical particles are mixed, and the proper way to…
This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The discussion emphasizes the pragmatic elements in the derivation. An epistemic notion of information is defined in…
The probability axioms by R. T. Cox can be regarded as the modern foundations of Bayesian inference, the idea of assigning degrees of belief to logical propositions in a manner consistent with Boolean logic. In this work it is shown that…
The progress of machine learning over the past decade is undeniable. In retrospect, it is both remarkable and unsettling that this progress was achievable with little to no rigorous theory to guide experimentation. Despite this fact,…
A message of any sort can be regarded as a source of information. Claude. E. Shannon showed in the last century that information ("what we don't already know") is equivalent to the entropy as defined in statistical mechanics. A string of…