English

Unlocking FedNL: Self-Contained Compute-Optimized Implementation

Machine Learning 2024-12-13 v2 Artificial Intelligence Mathematical Software Performance Optimization and Control

Abstract

Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner, eliminating the need for sharing their local data. The recent work (arXiv:2106.02969) introduces a family of Federated Newton Learn (FedNL) algorithms, marking a significant step towards applying second-order methods to FL and large-scale optimization. However, the reference FedNL prototype exhibits three serious practical drawbacks: (i) It requires 4.8 hours to launch a single experiment in a sever-grade workstation; (ii) The prototype only simulates multi-node setting; (iii) Prototype integration into resource-constrained applications is challenging. To bridge the gap between theory and practice, we present a self-contained implementation of FedNL, FedNL-LS, FedNL-PP for single-node and multi-node settings. Our work resolves the aforementioned issues and reduces the wall clock time by x1000. With this FedNL outperforms alternatives for training logistic regression in a single-node -- CVXPY (arXiv:1603.00943), and in a multi-node -- Apache Spark (arXiv:1505.06807), Ray/Scikit-Learn (arXiv:1712.05889). Finally, we propose two practical-orientated compressors for FedNL - adaptive TopLEK and cache-aware RandSeqK, which fulfill the theory of FedNL.

Keywords

Cite

@article{arxiv.2410.08760,
  title  = {Unlocking FedNL: Self-Contained Compute-Optimized Implementation},
  author = {Konstantin Burlachenko and Peter Richtárik},
  journal= {arXiv preprint arXiv:2410.08760},
  year   = {2024}
}

Comments

55 pages, 12 figures, 12 tables

R2 v1 2026-06-28T19:17:45.465Z