English

SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication

Machine Learning 2026-05-11 v1 Artificial Intelligence Distributed, Parallel, and Cluster Computing

Abstract

In large-scale reinforcement learning (RL) systems with decoupled Trainer-Rollout execution, the Trainer must regularly synchronize policy weights to the Rollout side to limit policy staleness. When inter-node bandwidth is abundant, such synchronization is usually only a small fraction of end-to-end cost. As model size grows, however, the communication demand rises rapidly. In bandwidth-constrained or network-variable deployments -- for example, cross-datacenter or cross-cluster settings, heterogeneous resource pools, and online RL -- weight synchronization can become a dominant bottleneck for throughput and tail latency. We observe that, in mainstream large-model RL training, the locations where parameters actually change are highly sparse at the element level (often 99%+ sparsity). Building on this observation, we propose and implement SparseRL-Sync, which replaces full-weight transfers with a lossless sparse update payload (indices and values) that can be exactly reconstructed on the inference side, thereby preserving 100% fidelity. Under a simplified cost model, sparse synchronization reduces the per-update communication volume from S to approximately S/X; with 99% sparsity (X ~ 100), this yields about a 100x reduction in transmitted data. Combined with appropriate bucketing, SparseRL-Sync also reduces launch and control-plane overhead, significantly improving scalability and end-to-end efficiency in bandwidth-limited and highly asynchronous RL settings.

Keywords

Cite

@article{arxiv.2605.07330,
  title  = {SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication},
  author = {Lucas Hu and Ranchi Zhao and Isaac Zhu and Zach Zhang and Hscos Zhang and Hugh Yin and Jason Zhao},
  journal= {arXiv preprint arXiv:2605.07330},
  year   = {2026}
}

Comments

Code will be released at https://github.com/scitix/helix

R2 v1 2026-07-01T12:57:02.491Z