Related papers: A Bayesian Characterization of Relative Entropy
Finite heat reservoir capacity and temperature fluctuations lead to modification of the well known canonical exponential weight factor. Requiring that the corrections least depend on the one-particle energy, we derive a deformed entropy,…
Estimating the Shannon entropy of a discrete distribution from which we have only observed a small sample is challenging. Estimating other information-theoretic metrics, such as the Kullback-Leibler divergence between two sparsely sampled…
We develop a maximum relative entropy formalism to generate optimal approximations to probability distributions. The central results consist in (a) justifying the use of relative entropy as the uniquely natural criterion to select a…
We consider the problem of estimating the population probability distribution given a finite set of multivariate samples, using the maximum entropy approach. In strict keeping with Jaynes' original definition, our precise formulation of the…
In this article, we construct semiparametrically efficient estimators of linear functionals of a probability measure in the presence of side information using an easy empirical likelihood approach. We use estimated constraint functions and…
The relationship between three probability distributions and their maximizable entropy forms is discussed without postulating entropy property. For this purpose, the entropy I is defined as a measure of uncertainty of the probability…
A well-known result across information theory, machine learning, and statistical physics shows that the maximum entropy distribution under a mean constraint has an exponential form called the Gibbs-Boltzmann distribution. This is used for…
Kullback-Leibler relative-entropy, in cases involving distributions resulting from relative-entropy minimization, has a celebrated property reminiscent of squared Euclidean distance: it satisfies an analogue of the Pythagoras' theorem. And…
We present theoretical properties of the log-concave maximum likelihood estimator of a density based on an independent and identically distributed sample in $\mathbb{R}^d$. Our study covers both the case where the true underlying density is…
Works, briefly surveyed here, are concerned with two basic methods: Maximum Probability and Bayesian Maximum Probability; as well as with their asymptotic instances: Relative Entropy Maximization and Maximum Non-parametric Likelihood.…
This work proposes a view of probability as a relative measure rather than an absolute one. To demonstrate this concept, we focus on finite outcome spaces and develop three fundamental axioms that establish requirements for relative…
A fundamental problem arising in many areas of machine learning is the evaluation of the likelihood of a given observation under different nominal distributions. Frequently, these nominal distributions are themselves estimated from data,…
The concept of Entropy plays a key role in Information Theory, Statistics, and Machine Learning.This paper introduces a new entropy measure, called the t-entropy, which exploits the concavity of the inverse-tan function. We analytically…
We present a new lower bound on the differential entropy rate of stationary processes whose sequences of probability density functions fulfill certain regularity conditions. This bound is obtained by showing that the gap between the…
We propose a general algorithm for approximating nonstandard Bayesian posterior distributions. The algorithm minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribution. Our method…
This thesis synthesizes probability and entropic inference with Quantum Mechanics (QM) and quantum measurement [1-6]. It is shown that the standard and quantum relative entropies are tools designed for the purpose of updating probability…
Projection Pursuit methodology permits to solve the difficult problem of finding an estimate of a density defined on a set of very large dimension. In his seminal article, Huber (see "Projection pursuit", Annals of Statistics, 1985)…
We analyze dynamical properties of the Shannon information entropy of a continuous probability distribution, which is driven by a standard diffusion process. This entropy choice is confronted with another option, employing the conditional…
We examine the fundamental aspects of statistical mechanics, dividing the problem into a discussion purely about probability, which we analyse from a Bayesian standpoint. We argue that the existence of a unique maximising probability…
We examine the {combinatorial} or {probabilistic} definition ("Boltzmann's principle") of the entropy or cross-entropy function $H \propto \ln \mathbb{W}$ or $D \propto - \ln \mathbb{P}$, where $\mathbb{W}$ is the statistical weight and…