Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent
Machine Learning
2021-07-05 v3 Optimization and Control
Machine Learning
Abstract
We present a primal only derivation of Mirror Descent as a "partial" discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential. We contrast this discretization to Natural Gradient Descent, which is obtained by a "full" forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to general Riemannian geometries, even when the metric tensor is {\em not} a Hessian, and thus there is no "dual."
Keywords
Cite
@article{arxiv.2004.01025,
title = {Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent},
author = {Suriya Gunasekar and Blake Woodworth and Nathan Srebro},
journal= {arXiv preprint arXiv:2004.01025},
year = {2021}
}
Comments
11 pages