Related papers: Lectures on Probability, Entropy, and Statistical …
One of the goals of probabilistic inference is to decide whether an empirically observed distribution is compatible with a candidate Bayesian network. However, Bayesian networks with hidden variables give rise to highly non-trivial…
The intuitive reasoning of physicists in conditions of uncertainty is closer to the Bayesian approach than to the frequentist ideas taught at University and which are considered the reference framework for handling statistical problems. The…
Information accounting provides a better foundation for hypothesis testing than does uncertainty quantification. A quantitative account of science is derived under this perspective that alleviates the need for epistemic bridge principles,…
We try to establish a unified information theoretic approach to learning and to explore some of its applications. First, we define {\em predictive information} as the mutual information between the past and the future of a time series,…
In the 21st century, many of the crucial scientific and technical issues facing humanity can be understood as problems associated with understanding, modelling, and ultimately controlling complex systems: systems comprised of a large number…
A colloquial interpretation of entropy is that it is the knowledge gained upon learning the outcome of a random experiment. Conditional entropy is then interpreted as the knowledge gained upon learning the outcome of one random experiment…
Inferring models, predicting the future, and estimating the entropy rate of discrete-time, discrete-event processes is well-worn ground. However, a much broader class of discrete-event processes operates in continuous-time. Here, we provide…
What is the relationship between plausibility logic and the principle of maximum entropy? When does the principle give unreasonable or wrong results? When is it appropriate to use the rule `expectation = average'? Can plausibility logic…
The uncertainty principle can be expressed in entropic terms, also taking into account the role of entanglement in reducing uncertainty. The information exclusion principle bounds instead the correlations that can exist between the outcomes…
The quantification of aleatoric and epistemic uncertainty in terms of conditional entropy and mutual information, respectively, has recently become quite common in machine learning. While the properties of these measures, which are rooted…
We characterize information as risk reduction between knowledge states represented by partitions of the underlying probability space. Entropy corresponds to risk reduction from no (or partial) knowledge to full knowledge about a random…
The problem of induction has persisted since Hume exposed the logical gap between repeated observation and universal inference. Traditional attempts to resolve it have oscillated between two extremes: the probabilistic optimism of Laplace…
In recent years, there has been an increased need for the use of active systems - systems required to act automatically based on events, or changes in the environment. Such systems span many areas, from active databases to applications that…
This chapter provides a overview of Bayesian inference, mostly emphasising that it is a universal method for summarising uncertainty and making estimates and predictions using probability statements conditional on observed data and an…
A given question can be defined in terms of the set of statements or assertions that answer it. Application of logical inference to these sets of assertions allows one to derive the logic of inquiry among questions. There are interesting…
Entropy is the distinguishing and most important concept of our efforts to understand and regularize our observations of a very large class of natural phenomena, and yet, it is one of the most contentious concepts of physics. In this…
Automated reasoning about uncertain knowledge has many applications. One difficulty when developing such systems is the lack of a completely satisfactory integration of logic and probability. We address this problem directly. Expressive…
A message of any sort can be regarded as a source of information. Claude. E. Shannon showed in the last century that information ("what we don't already know") is equivalent to the entropy as defined in statistical mechanics. A string of…
Despite the wide usage of information as a concept in science, we have yet to develop a clear & concise scientific definition. This paper is aimed at laying the foundations for a new theory concerning the mechanics of information alongside…
During a spontaneous change, a macroscopic physical system will evolve towards a macro-state with more realizations. This observation is at the basis of the Statistical Mechanical version of the Second Law of Thermodynamics, and it provides…