Randomized Kaczmarz with geometrically smoothed momentum
Numerical Analysis
2024-08-27 v3 Numerical Analysis
Probability
Machine Learning
Abstract
This paper studies the effect of adding geometrically smoothed momentum to the randomized Kaczmarz algorithm, which is an instance of stochastic gradient descent on a linear least squares loss function. We prove a result about the expected error in the direction of singular vectors of the matrix defining the least squares loss. We present several numerical examples illustrating the utility of our result and pose several questions.
Cite
@article{arxiv.2401.09415,
title = {Randomized Kaczmarz with geometrically smoothed momentum},
author = {Seth J. Alderman and Roan W. Luikart and Nicholas F. Marshall},
journal= {arXiv preprint arXiv:2401.09415},
year = {2024}
}
Comments
28 pages, 16 figures