A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information
Optimization and Control
2022-08-24 v1 Machine Learning
Machine Learning
Abstract
In this paper, we consider to improve the stochastic variance reduce gradient (SVRG) method via incorporating the curvature information of the objective function. We propose to reduce the variance of stochastic gradients using the computationally efficient Barzilai-Borwein (BB) method by incorporating it into the SVRG. We also incorporate a BB-step size as its variant. We prove its linear convergence theorem that works not only for the proposed method but also for the other existing variants of SVRG with second-order information. We conduct the numerical experiments on the benchmark datasets and show that the proposed method with constant step size performs better than the existing variance reduced methods for some test problems.
Cite
@article{arxiv.2208.11075,
title = {A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information},
author = {Hardik Tankaria and Nobuo Yamashita},
journal= {arXiv preprint arXiv:2208.11075},
year = {2022}
}