On the Natural Gradient of the Evidence Lower Bound
Abstract
This article studies the Fisher-Rao gradient, also referred to as the natural gradient, of the evidence lower bound (ELBO) which plays a central role in generative machine learning. It reveals that the gap between the evidence and its lower bound, the ELBO, has essentially a vanishing natural gradient within unconstrained optimization. As a result, maximization of the ELBO is equivalent to minimization of the Kullback-Leibler divergence from a target distribution, the primary objective function of learning. Building on this insight, we derive a condition under which this equivalence persists even when optimization is constrained to a model. This condition yields a geometric characterization, which we formalize through the notion of a cylindrical model.
Keywords
Cite
@article{arxiv.2307.11249,
title = {On the Natural Gradient of the Evidence Lower Bound},
author = {Nihat Ay and Jesse van Oostrum and Adwait Datar},
journal= {arXiv preprint arXiv:2307.11249},
year = {2025}
}