English

Unveiling High-Probability Generalization in Decentralized SGD

Machine Learning 2026-05-12 v1

Abstract

Decentralized stochastic gradient descent (D-SGD) is an efficient method for large-scale distributed learning. Existing generalization studies mainly address expected results, achieving rates limited to O(1δmn)\mathcal{O}\left(\frac{1}{\delta \sqrt{mn}}\right), where δ\delta is the confidence parameter, mm the number of workers, and nn the sample size. When m=1m=1, D-SGD reduces to traditional SGD, whose optimal high-probability generalization bound is O(1nlog(1/δ))\mathcal{O}\left(\frac{1}{\sqrt{n}}\log (1/\delta)\right). This discrepancy reveals a gap between high-probability guarantees for SGD and those for D-SGD. To close this, we develop a high-probability learning theory for D-SGD, aiming for the optimal O(1mnlog(1/δ))\mathcal{O}\left(\frac{1}{\sqrt{mn}}\log (1/\delta)\right) rate. We refine bounds for D-SGD using pointwise uniform stability in distributed learning-a weaker notion than uniform stability-and analyze them across convex, strongly convex, and non-convex settings. We also provide high-probability results for gradient-based measures in non-convex cases where only local minima exist, and derive optimization error and excess risk bounds. Finally, accounting for communication overhead, we analyze generalization bounds for local models within time-varying frameworks.

Keywords

Cite

@article{arxiv.2605.10205,
  title  = {Unveiling High-Probability Generalization in Decentralized SGD},
  author = {Jiahuan Wang and Ping Luo and Ziqing Wen and Dongsheng Li and Tao Sun},
  journal= {arXiv preprint arXiv:2605.10205},
  year   = {2026}
}