English

FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission

Machine Learning 2024-03-05 v1 Distributed, Parallel, and Cluster Computing Networking and Internet Architecture

Abstract

Communication overhead is a significant bottleneck in federated learning (FL), which has been exaggerated with the increasing size of AI models. In this paper, we propose FedRDMA, a communication-efficient cross-silo FL system that integrates RDMA into the FL communication protocol. To overcome the limitations of RDMA in wide-area networks (WANs), FedRDMA divides the updated model into chunks and designs a series of optimization techniques to improve the efficiency and robustness of RDMA-based communication. We implement FedRDMA atop the industrial federated learning framework and evaluate it on a real-world cross-silo FL scenario. The experimental results show that \sys can achieve up to 3.8×\times speedup in communication efficiency compared to traditional TCP/IP-based FL systems.

Keywords

Cite

@article{arxiv.2403.00881,
  title  = {FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission},
  author = {Zeling Zhang and Dongqi Cai and Yiran Zhang and Mengwei Xu and Shangguang Wang and Ao Zhou},
  journal= {arXiv preprint arXiv:2403.00881},
  year   = {2024}
}

Comments

under review