Structured second-order methods via natural gradient descent
Machine Learning
2022-02-22 v3 Machine Learning
Abstract
In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces. Natural-gradient descent is an attractive approach to design new algorithms in many settings such as gradient-free, adaptive-gradient, and second-order methods. Our structured methods not only enjoy a structural invariance but also admit a simple expression. Finally, we test the efficiency of our proposed methods on both deterministic non-convex problems and deep learning problems.
Cite
@article{arxiv.2107.10884,
title = {Structured second-order methods via natural gradient descent},
author = {Wu Lin and Frank Nielsen and Mohammad Emtiyaz Khan and Mark Schmidt},
journal= {arXiv preprint arXiv:2107.10884},
year = {2022}
}
Comments
Fixed some typos and added a new figure. ICML 2021 workshop paper. A short version of arXiv:2102.07405 with a focus on optimization tasks