Natural Gradient for Combined Loss Using Wavelets
Numerical Analysis
2020-06-30 v1 Machine Learning
Numerical Analysis
Optimization and Control
Abstract
Natural gradients have been widely used in optimization of loss functionals over probability space, with important examples such as Fisher-Rao gradient descent for Kullback-Leibler divergence, Wasserstein gradient descent for transport-related functionals, and Mahalanobis gradient descent for quadratic loss functionals. This note considers the situation in which the loss is a convex linear combination of these examples. We propose a new natural gradient algorithm by utilizing compactly supported wavelets to diagonalize approximately the Hessian of the combined loss. Numerical results are included to demonstrate the efficiency of the proposed algorithm.
Cite
@article{arxiv.2006.15806,
title = {Natural Gradient for Combined Loss Using Wavelets},
author = {Lexing Ying},
journal= {arXiv preprint arXiv:2006.15806},
year = {2020}
}