Related papers: On the probabilistic rationale of I-divergence and…
This paper modifies Jaynes's axioms of plausible reasoning and derives the minimum relative entropy principle, Bayes's rule, as well as maximum likelihood from first principles. The new axioms, which I call the Optimum Information…
Works, briefly surveyed here, are concerned with two basic methods: Maximum Probability and Bayesian Maximum Probability; as well as with their asymptotic instances: Relative Entropy Maximization and Maximum Non-parametric Likelihood.…
The relationship between three probability distributions and their maximizable entropy forms is discussed without postulating entropy property. For this purpose, the entropy I is defined as a measure of uncertainty of the probability…
We develop and analyze $M$-estimation methods for divergence functionals and the likelihood ratios of two probability distributions. Our method is based on a non-asymptotic variational characterization of $f$-divergences, which allows the…
The extremization of an appropriate entropic functional may yield to the probability distribution functions maximizing the respective entropic structure. This procedure is known in Statistical Mechanics and Information Theory as Jaynes'…
We develop a maximum relative entropy formalism to generate optimal approximations to probability distributions. The central results consist in (a) justifying the use of relative entropy as the uniquely natural criterion to select a…
Minimization problems with respect to a one-parameter family of generalized relative entropies are studied. These relative entropies, which we term relative $\alpha$-entropies (denoted $\mathscr{I}_{\alpha}$), arise as redundancies under…
The maximum entropy formalism developed by Jaynes determines the relevant ensemble in nonequilibrium statistical mechanics by maximising the entropy functional subject to the constraints imposed by the available information. We present an…
We study minimization of a parametric family of relative entropies, termed relative $\alpha$-entropies (denoted $\mathscr{I}_{\alpha}(P,Q)$). These arise as redundancies under mismatched compression when cumulants of compressed lengths are…
The problem of determining the joint probability distributions for correlated random variables with pre-specified marginals is considered. When the joint distribution satisfying all the required conditions is not unique, the "most unbiased"…
Entropy maximization procedure has been a general practice in many diverse fields of science to obtain the concomitant probability distributions. The consistent use of the maximization procedure on the other hand requires the probability…
We develop a new semantics for defeasible inference based on extended probability measures allowed to take infinitesimal values, on the interpretation of defaults as generalized conditional probability constraints and on a preferred-model…
The fundamentals of the Maximum Entropy principle as a rule for assigning and updating probabilities are revisited. The Shannon-Jaynes relative entropy is vindicated as the optimal criterion for use with an updating rule. A constructive…
The equilibrium distributions of probabilities providing maximality of Renyi and Tsallis entropies are rederived. New S-forms of them are found which are normalised with corresponding entropies in contrast to the usual Z-forms normalised…
This paper develops a new divergence that generalizes relative entropy and can be used to compare probability measures without a requirement of absolute continuity. We establish properties of the divergence, and in particular derive and…
Maximum entropy approach to classification is very well studied in applied statistics and machine learning and almost all the methods that exists in literature are discriminative in nature. In this paper, we introduce a maximum entropy…
Probabilistic reasoning systems combine different probabilistic rules and probabilistic facts to arrive at the desired probability values of consequences. In this paper we describe the MESA-algorithm (Maximum Entropy by Simulated Annealing)…
Given two discrete random variables $X$ and $Y$, with probability distributions ${\bf p} =(p_1, \ldots , p_n)$ and ${\bf q}=(q_1, \ldots , q_m)$, respectively, denote by ${\cal C}({\bf p}, {\bf q})$ the set of all couplings of ${\bf p}$ and…
We take another look at the general problem of selecting a preferred probability measure among those that comply with some given constraints. The dominant role that entropy maximization has obtained in this context is questioned by arguing…
The optimality and sensitivity of the empirical risk minimization problem with relative entropy regularization (ERM-RER) are investigated for the case in which the reference is a sigma-finite measure instead of a probability measure. This…