Related papers: Information and Entropy
We examine the relationship between the Bayesian and information-theoretic formulations of source separation algorithms. This work makes use of the relationship between the work of Claude E. Shannon and the "Recent Contributions" by Warren…
Persuasion is the process of changing an agent's belief distribution from a given (or estimated) prior to a desired posterior. A common assumption in the acceptance of information or misinformation as fact is that the (mis)information must…
This chapter provides a overview of Bayesian inference, mostly emphasising that it is a universal method for summarising uncertainty and making estimates and predictions using probability statements conditional on observed data and an…
An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output…
In these decades, it has been revealed that there is rich information-theoretic structure in thermodynamics of out-of-equilibrium systems in both the classical and quantum regimes. This has led to the fruitful interplay among statistical…
We define the concept of dependence among multiple variables using maximum entropy techniques and introduce a graphical notation to denote the dependencies. Direct inference of information theoretic quantities from data uncovers…
Information theory is an outstanding framework to measure uncertainty, dependence and relevance in data and systems. It has several desirable properties for real world applications: it naturally deals with multivariate data, it can handle…
A message of any sort can be regarded as a source of information. Claude. E. Shannon showed in the last century that information ("what we don't already know") is equivalent to the entropy as defined in statistical mechanics. A string of…
In inverse problems, one attempts to infer spatially variable functions from indirect measurements of a system. To practitioners of inverse problems, the concept of "information" is familiar when discussing key questions such as which parts…
Motivated reasoning posits that people distort how they process information in the direction of beliefs they find attractive. This paper creates a novel experimental design to identify motivated reasoning from Bayesian updating when people…
The concept of information has emerged as a language in its own right, bridging several disciplines that analyze natural phenomena and man-made systems. Integrated information has been introduced as a metric to quantify the amount of…
The concept of free energy has its origins in 19th century thermodynamics, but has recently found its way into the behavioral and neural sciences, where it has been promoted for its wide applicability and has even been suggested as a…
Entropy Search (ES) and Predictive Entropy Search (PES) are popular and empirically successful Bayesian Optimization techniques. Both rely on a compelling information-theoretic motivation, and maximize the information gained about the…
Probability theory, epistemically interpreted, provides an excellent, if not the best available account of inductive reasoning. This is so because there are general and definite rules for the change of subjective probabilities through…
Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these…
Information based thermodynamic logic is revisited. It consists of two parts: Part A applies the modern theory of probability in which an arbitrary convex function \phi is employed as an analytic "device" to express information as…
Maximum entropy estimation is of broad interest for inferring properties of systems across many different disciplines. In this work, we significantly extend a technique we previously introduced for estimating the maximum entropy of a set of…
Entropy is a measure of self-information which is used to quantify losses. Entropy was developed in thermodynamics, but is also used to compare probabilities based on their deviating information content. Corresponding model uncertainty is…
Bayesian hierarchical models are frequently used in practical data analysis contexts. One interpretation of these models is that they provide an indirect way of assigning a prior for unknown parameters, through the introduction of…
Information theory, though originally developed for communications engineering, provides mathematical tools with broad applications across science. These tools characterize the fundamental limits of data compression and transmission in the…