Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds
Abstract
In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. Since matrices and tensors of fixed rank form smooth Riemannian manifolds, one of the popular tools for finding low-rank approximations is to use Riemannian optimization. Nevertheless, efficient implementation of Riemannian gradients and Hessians, required in Riemannian optimization algorithms, can be a nontrivial task in practice. Moreover, in some cases, analytic formulas are not even available. In this paper, we build upon automatic differentiation and propose a method that, given an implementation of the function to be minimized, efficiently computes Riemannian gradients and matrix-by-vector products between an approximate Riemannian Hessian and a given vector.
Cite
@article{arxiv.2103.14974,
title = {Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds},
author = {Alexander Novikov and Maxim Rakhuba and Ivan Oseledets},
journal= {arXiv preprint arXiv:2103.14974},
year = {2021}
}