Memory-Efficient Split Federated Learning for LLM Fine-Tuning on Heterogeneous Mobile Devices

Xiaopei Chen; Liang Li; Fei Ji; Wen Wu

Memory-Efficient Split Federated Learning for LLM Fine-Tuning on Heterogeneous Mobile Devices

Distributed, Parallel, and Cluster Computing 2025-06-04 v1

Authors: Xiaopei Chen , Liang Li , Fei Ji , Wen Wu

Abstract

In this paper, we propose an edge-assisted split federated learning framework to facilitate large language model (LLM) fine-tuning on heterogeneous mobile devices while alleviating memory pressures on both mobile devices and the edge server. Specifically, mobile devices perform low-rank adaptation (LoRA) fine-tuning on only a subset of lower layers of the pre-trained LLM, tailored to their individual capacities. On the server, a full LLM is maintained, and the corresponding LoRA modules are selectively fine-tuned in a sequential manner for each device. To further enhance training efficiency, we propose a server-side training scheduling method that optimizes the processing order of devices for accelerating fine-tuning. Extensive experiments demonstrate that compared to the baselines, our scheme can reduce 79\% memory footprint and 6\% training time while achieving comparable performance.

Keywords

large language model training parameter-efficient fine-tuning large language model inference

Cite

@article{arxiv.2506.02940,
  title  = {Memory-Efficient Split Federated Learning for LLM Fine-Tuning on Heterogeneous Mobile Devices},
  author = {Xiaopei Chen and Liang Li and Fei Ji and Wen Wu},
  journal= {arXiv preprint arXiv:2506.02940},
  year   = {2025}
}

Comments

IEEE INFOCOM IEILM 2025

Memory-Efficient Split Federated Learning for LLM Fine-Tuning on Heterogeneous Mobile Devices

Abstract

Keywords

Cite

Comments

Related papers