Related papers: Information functionals and the notion of (un)cert…
Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these…
Exponential models of distributions are widely used in machine learning for classiffication and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we…
Information theory is built on probability measures and by definition a probability measure has total mass 1. Probability measures are used to model uncertainty, and one may ask how important it is that the total mass is one. We claim that…
In this paper, it is shown that the rank function of a matroid can be represented by a "mutual information function" if and only if the matroid is binary. The mutual information function considered is the one measuring the amount of…
We consider the analysis of probability distributions through their associated covariance operators from reproducing kernel Hilbert spaces. We show that the von Neumann entropy and relative entropy of these operators are intimately related…
Determining functionals are tools to describe the finite dimensional long-term dynamics of infinite dimensional dynamical systems. There also exist several applications to infinite dimensional {\em random} dynamical systems. In these…
The random matrix ensembles (RME), especially Gaussian random matrix ensembles GRME and Ginibre random matrix ensembles, are applied to following quantum systems: nuclear systems, molecular systems, and two-dimensional electron systems…
Belief functions are a powerful and popular framework for the mathematical characterisation of uncertainty, in particular in situations in which lack of data renders learning a probability distribution for the problem impractical. The first…
We develop and analyze $M$-estimation methods for divergence functionals and the likelihood ratios of two probability distributions. Our method is based on a non-asymptotic variational characterization of $f$-divergences, which allows the…
Entropy is a measure of self-information which is used to quantify losses. Entropy was developed in thermodynamics, but is also used to compare probabilities based on their deviating information content. Corresponding model uncertainty is…
Entropy-type integral functionals of densities are widely used in mathematical statistics, information theory, and computer science. Examples include measures of closeness between distributions (e.g., density power divergence) and…
In machine learning or scientific computing, model performance is measured with an objective function. But why choose one objective over another? Information theory gives one answer: To maximize the information in the model, select the…
A new method for clustering functional data is proposed via information maximization. The proposed method learns a probabilistic classifier in an unsupervised manner so that mutual information (or squared loss mutual information) between…
Random matrix theory (RMT) is based on two assumptions: (1) matrix-element independence, and (2) base invariance. Most of the proposed generalizations keep the first assumption and violate the second. Recently, several authors presented…
We consider an estimation problem of expected functionals of a general random element that values in a metric space. If the functional forms an explicit function of some unknown parameters, we can estimate it by plugging-in a suitable…
This paper presents a new foundational approach to information theory based on the concept of the information efficiency of a recursive function, which is defined as the difference between the information in the input and the output. The…
Despite the wide usage of information as a concept in science, we have yet to develop a clear & concise scientific definition. This paper is aimed at laying the foundations for a new theory concerning the mechanics of information alongside…
Information projections are the key building block of variational inference algorithms and are used to approximate a target probabilistic model by projecting it onto a family of tractable distributions. In general, there is no guarantee on…
Interpretability is central to trustworthy machine learning, yet existing metrics rarely quantify how effectively data support an interpretive representation. We propose Interpretive Efficiency, a normalized, task-aware functional that…
We take another look at the general problem of selecting a preferred probability measure among those that comply with some given constraints. The dominant role that entropy maximization has obtained in this context is questioned by arguing…