Revisiting Ensembling in One-Shot Federated Learning
Abstract
Federated learning (FL) is an appealing approach to training machine learning models without sharing raw data. However, standard FL algorithms are iterative and thus induce a significant communication cost. One-shot federated learning (OFL) trades the iterative exchange of models between clients and the server with a single round of communication, thereby saving substantially on communication costs. Not surprisingly, OFL exhibits a performance gap in terms of accuracy with respect to FL, especially under high data heterogeneity. We introduce FENS, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL. Learning in FENS proceeds in two phases: first, clients train models locally and send them to the server, similar to OFL; second, clients collaboratively train a lightweight prediction aggregator model using FL. We showcase the effectiveness of FENS through exhaustive experiments spanning several datasets and heterogeneity levels. In the particular case of heterogeneously distributed CIFAR-10 dataset, FENS achieves up to a 26.9% higher accuracy over state-of-the-art (SOTA) OFL, being only 3.1% lower than FL. At the same time, FENS incurs at most 4.3x more communication than OFL, whereas FL is at least 10.9x more communication-intensive than FENS.
Cite
@article{arxiv.2411.07182,
title = {Revisiting Ensembling in One-Shot Federated Learning},
author = {Youssef Allouah and Akash Dhasade and Rachid Guerraoui and Nirupam Gupta and Anne-Marie Kermarrec and Rafael Pinot and Rafael Pires and Rishi Sharma},
journal= {arXiv preprint arXiv:2411.07182},
year = {2024}
}
Comments
Accepted at NeurIPS 2024