SPAN: A Stochastic Projected Approximate Newton Method
Abstract
Second-order optimization methods have desirable convergence properties. However, the exact Newton method requires expensive computation for the Hessian and its inverse. In this paper, we propose SPAN, a novel approximate and fast Newton method. SPAN computes the inverse of the Hessian matrix via low-rank approximation and stochastic Hessian-vector products. Our experiments on multiple benchmark datasets demonstrate that SPAN outperforms existing first-order and second-order optimization methods in terms of the convergence wall-clock time. Furthermore, we provide a theoretical analysis of the per-iteration complexity, the approximation error, and the convergence rate. Both the theoretical analysis and experimental results show that our proposed method achieves a better trade-off between the convergence rate and the per-iteration efficiency.
Cite
@article{arxiv.2002.03687,
title = {SPAN: A Stochastic Projected Approximate Newton Method},
author = {Xunpeng Huang and Xianfeng Liang and Zhengyang Liu and Yitan Li and Linyun Yu and Yue Yu and Lei Li},
journal= {arXiv preprint arXiv:2002.03687},
year = {2020}
}
Comments
Appeared in the AAAI 2020, 25 pages, 6 figures