English

The convergence of the Stochastic Gradient Descent (SGD) : a self-contained proof

Machine Learning 2023-11-15 v2 Machine Learning Probability
Authors: Gabrel Turinici
View on arXiv ↗ PDF ↗ DOI ↗

Abstract

We give here a proof of the convergence of the Stochastic Gradient Descent (SGD) in a self-contained manner.

Keywords

stochastic gradient descent convergence self-contained proof

Cite

@article{arxiv.2103.14350,
  title  = {The convergence of the Stochastic Gradient Descent (SGD) : a self-contained proof},
  author = {Gabrel Turinici},
  journal= {arXiv preprint arXiv:2103.14350},
  year   = {2023}
}

Related papers

View all related →
Optimization and Control · Mathematics
Stochastic Gradient Descent Revisited
Azar Louzi
2025-03-11
Optimization and Control · Mathematics
A Stochastic Gradient Descent Theorem and the Back-Propagation Algorithm
Hao Wu
2021-04-02
Machine Learning · Computer Science
Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum
Jerry Chee, Ping Li
2020-08-28
Optimization and Control · Mathematics
Convergence rates and approximation results for SGD and its continuous-time counterpart
Xavier Fontaine, Valentin De Bortoli, Alain Durmus
2021-02-02
Machine Learning · Computer Science
Stochastic Gradient Descent with Biased but Consistent Gradient Estimators
Jie Chen, Ronny Luss
2019-12-24
Machine Learning · Computer Science
A Parallel SGD method with Strong Convergence
Dhruv Mahajan, S. Sathiya Keerthi, S. Sundararajan, Leon Bottou
2013-11-05
Machine Learning · Computer Science
Convergence Analysis of SGD under Expected Smoothness
Yuta Kawamoto, Hideaki Iiduka
2025-10-28
Machine Learning · Statistics
Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
Mor Shpigel Nacson, Nathan Srebro, Daniel Soudry
2022-04-19
Optimization and Control · Mathematics
Gradient Descent-Type Methods: Background and Simple Unified Convergence Analysis
Quoc Tran-Dinh, Marten van Dijk
2022-12-20
Numerical Analysis · Mathematics
Batched Stochastic Gradient Descent with Weighted Sampling
Deanna Needell, Rachel Ward
2017-03-02
Statistics Theory · Mathematics
Asymptotic Analysis of Conditioned Stochastic Gradient Descent
Rémi Leluc, François Portier
2023-10-17
Machine Learning · Computer Science
Global Convergence and Stability of Stochastic Gradient Descent
Vivak Patel, Shushu Zhang, Bowen Tian
2022-10-11
Optimization and Control · Mathematics
Analysis of Biased Stochastic Gradient Descent Using Sequential Semidefinite Programs
Bin Hu, Peter Seiler, Laurent Lessard
2020-03-31
Numerical Analysis · Mathematics
Eigen-componentwise convergence of SGD on quadratic programming
Lehan Chen, Yuji Nakatsukasa
2024-12-11
Machine Learning · Computer Science
Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
Alexander Rakhlin, Ohad Shamir, Karthik Sridharan
2015-03-19
Optimization and Control · Mathematics
A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent
Eduard Gorbunov, Filip Hanzely, Peter Richtárik
2019-05-28
Machine Learning · Computer Science
On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants
Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos +1
2016-01-26
Optimization and Control · Mathematics
Finite-Time Analysis of Stochastic Gradient Descent under Markov Randomness
Thinh T. Doan, Lam M. Nguyen, Nhan H. Pham, Justin Romberg
2020-04-02
Optimization and Control · Mathematics
Stochastic gradient with least-squares control variates
Fabio Nobile, Matteo Raviola, Nathan Schaeffer
2025-11-21
Numerical Analysis · Mathematics
On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems
Zehui Zhou
2024-09-30
Machine Learning · Computer Science
Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions
Yunwen Lei, Ting Hu, Guiying Li, Ke Tang
2019-12-16
Machine Learning · Computer Science
Unified Optimal Analysis of the (Stochastic) Gradient Method
Sebastian U. Stich
2019-12-24
Machine Learning · Computer Science
Demystifying the Myths and Legends of Nonconvex Convergence of SGD
Aritra Dutta, El Houcine Bergou, Soumia Boucherouite, Nicklas Werge +2
2023-10-20
Optimization and Control · Mathematics
Error Lower Bounds of Constant Step-size Stochastic Gradient Descent
Zhiyan Ding, Yiding Chen, Qin Li, Xiaojin Zhu
2019-10-21
Optimization and Control · Mathematics
Handbook of Convergence Theorems for (Stochastic) Gradient Methods
Guillaume Garrigos, Robert M. Gower
2024-03-12