Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise
Abstract
In this paper, we propose practical normalized stochastic first-order methods with Polyak momentum, multi-extrapolated momentum, and recursive momentum for solving unconstrained optimization problems. These methods employ dynamically updated algorithmic parameters and do not require explicit knowledge of problem-dependent quantities such as the Lipschitz constant or noise bound. We establish first-order oracle complexity results for finding approximate stochastic stationary points under heavy-tailed noise and weakly average smoothness conditions -- both of which are weaker than the commonly used bounded variance and mean-squared smoothness assumptions. Our complexity bounds either improve upon or match the best-known results in the literature. Numerical experiments are presented to demonstrate the practical effectiveness of the proposed methods.
Cite
@article{arxiv.2506.11214,
title = {Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise},
author = {Chuan He and Zhaosong Lu and Defeng Sun and Zhanwang Deng},
journal= {arXiv preprint arXiv:2506.11214},
year = {2026}
}