Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple example
Machine Learning
2023-08-03 v1
Abstract
A very simple unidimensional function with Lipschitz continuous gradient is constructed such that the ADAM algorithm with constant stepsize, started from the origin, diverges when applied to minimize this function in the absence of noise on the gradient. Divergence occurs irrespective of the choice of the method parameters.
Keywords
Cite
@article{arxiv.2308.00720,
title = {Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple example},
author = {Ph. L. Toint},
journal= {arXiv preprint arXiv:2308.00720},
year = {2023}
}