Related papers: Information Processing Equalities and the Informat…
In many applications we have both observational and (randomized) interventional data. We propose a Gaussian likelihood framework for joint modeling of such different data-types, based on global parameters consisting of a directed acyclic…
Symmetric alpha-stable (S alpha S) distributions with alpha<2 lack finite classical Fisher information. Building on Johnson's framework, we define Mixed Fractional Information (MFI) via the initial rate of relative entropy dissipation…
This paper studies semiparametric Fisher information in models parametrized by general normed spaces. The main contribution is to establish that positive semiparametric Fisher information is equivalent to the gradient of the parameter of…
The problem of f-divergence estimation is important in the fields of machine learning, information theory, and statistics. While several nonparametric divergence estimators exist, relatively few have known convergence properties. In…
Classical inequalities used in information theory such as those of de Bruijn, Fisher, and Kullback carry over from the setting of probability theory on Euclidean space to that of unimodular Lie groups. These are groups that posses…
We introduce a new definition of exponential family of Markov chains, and show that many characteristic properties of the usual exponential family of probability distributions are properly extended to Markov chains. The method of…
$f$-divergences are a general class of divergences between probability measures which include as special cases many commonly used divergences in probability, mathematical statistics and information theory such as Kullback-Leibler…
In this work, we study generalized entropies and information geometry in a group-theoretical framework. We explore the conditions that ensure the existence of some natural properties and at the same time of a group-theoretical structure for…
The ability of machine learning (ML) algorithms to generalize well to unseen data has been studied through the lens of information theory, by bounding the generalization error with the input-output mutual information (MI), i.e., the MI…
This paper studies a Markov network model for unbalanced data, aiming to solve the problems of classification bias and insufficient minority class recognition ability of traditional machine learning models in environments with uneven class…
This report introduces general ideas and some basic methods of the Bayesian probability theory applied to physics measurements. Our aim is to make the reader familiar, through examples rather than rigorous formalism, with concepts such as:…
In this Note we introduce a new methodology for Bayesian inference through the use of $\phi$-divergences and the duality technique. The asymptotic laws of the estimates are established.
Mutual information I in infinite sequences (and in their finite prefixes) is essential in theoretical analysis of many situations. Yet its right definition has been elusive for a long time. I address it by generalizing Kolmogorov Complexity…
We present new algorithms for computing and approximating bisimulation metrics in Markov Decision Processes (MDPs). Bisimulation metrics are an elegant formalism that capture behavioral equivalence between states and provide strong…
We study the statistical complexity of estimating partition functions given sample access to a proposal distribution and an unnormalized density ratio for a target distribution. While partition function estimation is a classical problem,…
Markov models lie at the interface between statistical independence in a probability distribution and graph separation properties. We review model selection and estimation in directed and undirected Markov models with Gaussian…
Bayesian models have become very popular over the last years in several fields such as signal processing, statistics, and machine learning. Bayesian inference requires the approximation of complicated integrals involving posterior…
Information-theory based variational principles have proven effective at providing scalable uncertainty quantification (i.e. robustness) bounds for quantities of interest in the presence of nonparametric model-form uncertainty. In this…
Group-invariant probability distributions appear in many data-generative models in machine learning, such as graphs, point clouds, and images. In practice, one often needs to estimate divergences between such distributions. In this work, we…
We define a metric and a family of $\alpha$-connections in statistical manifolds, based on $\varphi$-divergence, which emerges in the framework of $\varphi$-families of probability distributions. This metric and $\alpha$-connections…