Unveiling High-Probability Generalization in Decentralized SGD
Abstract
Decentralized stochastic gradient descent (D-SGD) is an efficient method for large-scale distributed learning. Existing generalization studies mainly address expected results, achieving rates limited to , where is the confidence parameter, the number of workers, and the sample size. When , D-SGD reduces to traditional SGD, whose optimal high-probability generalization bound is . This discrepancy reveals a gap between high-probability guarantees for SGD and those for D-SGD. To close this, we develop a high-probability learning theory for D-SGD, aiming for the optimal rate. We refine bounds for D-SGD using pointwise uniform stability in distributed learning-a weaker notion than uniform stability-and analyze them across convex, strongly convex, and non-convex settings. We also provide high-probability results for gradient-based measures in non-convex cases where only local minima exist, and derive optimization error and excess risk bounds. Finally, accounting for communication overhead, we analyze generalization bounds for local models within time-varying frameworks.
Cite
@article{arxiv.2605.10205,
title = {Unveiling High-Probability Generalization in Decentralized SGD},
author = {Jiahuan Wang and Ping Luo and Ziqing Wen and Dongsheng Li and Tao Sun},
journal= {arXiv preprint arXiv:2605.10205},
year = {2026}
}