Related papers: Relative Entropy and Inductive Inference

We show that the naive application of the maximum entropy principle can yield answers which depend on the level of description, i.e. the result is not invariant under coarse-graining. We demonstrate that the correct approach, even for…

We define an entropy based on a chosen governing probability distribution. If a certain kind of measurements follow such a distribution it also gives us a suitable scale to study it with. This scale will appear as a link function that is…

Two different approaches to dealing with probabilistic knowledge are examined -models and inductive inference. Examples of the first are: influence diagrams [1], Bayesian networks [2], log-linear models [3, 4]. Examples of the second are:…

We consider the problem of exact probabilistic inference for Union of Conjunctive Queries (UCQs) on tuple-independent databases. For this problem, two approaches currently coexist. In the extensional method, query evaluation is performed by…

Entropy estimation is of practical importance in information theory and statistical science. Many existing entropy estimators suffer from fast growing estimation bias with respect to dimensionality, rendering them unsuitable for…

Based on the q-exponential distribution which has been observed in more and more physical systems, the varentropy method is used to derive the uncertainty measure of such an abnormal distribution function. The uncertainty measure obtained…

How to find unknown distributions is questioned in many pieces of research. There are several ways to figure them out, but the main question is which acts more reasonably than others. In this paper, we focus on the maximum entropy principle…

The maximum entropy principle advocates to evaluate events' probabilities using a distribution that maximizes entropy among those that satisfy certain expectations' constraints. Such principle can be generalized for arbitrary decision…

Many complex systems are characterized by non-Boltzmann distribution functions of their statistical variables. If one wants to -- justified or not -- hold on to the maximum entropy principle for complex statistical systems (non-Boltzmann)…

We show, on purely statistical grounds and without appeal to any physical model, that a power-law $q-$entropy $S_q$, with $0<q<1$, can be {\it extensive}. More specifically, if the components $X_i$ of a vector $X \in \mathbb{R}^N$ are…

A well-known result across information theory, machine learning, and statistical physics shows that the maximum entropy distribution under a mean constraint has an exponential form called the Gibbs-Boltzmann distribution. This is used for…

There is a random variable (X) with a determined outcome (i.e., X = x0), p(x0) = 1. Consider x0 to have a discrete uniform distribution over the integer interval [1, s], where the size of the sample space (s) = 1, in the initial state, such…

The ability of many powerful machine learning algorithms to deal with large data sets without compromise is often hampered by computationally expensive linear algebra tasks, of which calculating the log determinant is a canonical example.…

We consider the maximum entropy problems associated with R\'enyi $Q$-entropy, subject to two kinds of constraints on expected values. The constraints considered are a constraint on the standard expectation, and a constraint on the…

The Renyi entropy with a free Renyi parameter $q$ is the most justified form of information entropy, and the Tsallis entropy may be regarded as a linear approximation to the Renyi entropy when $q\simeq 1$. When $q\to 1$, both entropies go…

The problem of constructing the $q=1/2$ non-extensive maximum entropy distributions from redundant and noisy data is considered. A strategy is proposed, which evolves through the following steps: i)independent constraints are first…

In density estimation task, maximum entropy model (Maxent) can effectively use reliable prior information via certain constraints, i.e., linear constraints without empirical parameters. However, reliable prior information is often…

Maximum Entropy (MaxEnt) reinforcement learning is a powerful learning paradigm which seeks to maximize return under entropy regularization. However, action entropy does not necessarily coincide with state entropy, e.g., when multiple…

We develop a maximum relative entropy formalism to generate optimal approximations to probability distributions. The central results consist in (a) justifying the use of relative entropy as the uniquely natural criterion to select a…

We propose a method to derive the stationary size distributions of a system, and the degree distributions of networks, using maximisation of the Gibbs-Shannon entropy. We apply this to a preferential attachment-type algorithm for systems of…