English

Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL

Machine Learning 2026-05-20 v2

Abstract

Bandwidth-constrained distributed reinforcement learning (RL) post-training of large language models is bottlenecked by two channels: weight synchronization from trainers to inference workers, and gradient or pseudo-gradient synchronization across trainers. We find that approximately 99% of per-step weight updates are invisible after the BF16 cast used by standard training and inference forward passes. We explain this sparsity by showing that, at typical RL post-training learning rates, Adam updates often fall below the local BF16 rounding threshold. We turn this observation into an algorithmic principle called compute-visible sparsification: transmit only updates that would change the next forward pass. PULSE (Precision-gated Updates for Low-precision Sparse Exchange) turns this principle into two communication algorithms: PULSESync sends lossless sparse BF16 weight patches from trainers to inference workers, and PULSELoCo sparsifies DiLoCo-style FP32 pseudo-gradient synchronization with error feedback. Over bandwidth-constrained commodity networks, PULSESync cuts weight-synchronization communication by over 100x while reconstructing trainer weights bit-identically. PULSELoCo matches DiLoCo across four models while reducing trainer-to-trainer communication by over 17x versus DiLoCo and over 100x versus DDP in the largest evaluated setting.

Keywords

Cite

@article{arxiv.2602.03839,
  title  = {Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL},
  author = {Erfan Miahi and Eugene Belilovsky},
  journal= {arXiv preprint arXiv:2602.03839},
  year   = {2026}
}

Comments

40 pages, 19 figures, 14 tables

R2 v1 2026-07-01T09:34:48.133Z