Gradient Descent Algorithm Survey
Machine Learning
2025-11-27 v1 Artificial Intelligence
Abstract
Focusing on the practical configuration needs of optimization algorithms in deep learning, this article concentrates on five major algorithms: SGD, Mini-batch SGD, Momentum, Adam, and Lion. It systematically analyzes the core advantages, limitations, and key practical recommendations of each algorithm. The research aims to gain an in-depth understanding of these algorithms and provide a standardized reference for the reasonable selection, parameter tuning, and performance improvement of optimization algorithms in both academic research and engineering practice, helping to solve optimization challenges in different scales of models and various training scenarios.
Cite
@article{arxiv.2511.20725,
title = {Gradient Descent Algorithm Survey},
author = {Deng Fucheng and Wang Wanjie and Gong Ao and Wang Xiaoqi and Wang Fan},
journal= {arXiv preprint arXiv:2511.20725},
year = {2025}
}