Training Mixture-of-Experts (MoE) models introduces sparse and highly imbalanced all-to-all communication that dominates iteration time. Conventional load-balancing methods fail to exploit the deterministic topology of Rail architectures, leaving multi-NIC bandwidth underutilized. We present RailS, a distributed load-balancing framework that minimizes all-to-all completion time in MoE training. RailS leverages the Rail topology's symmetry to prove that uniform sending ensures uniform receiving, transforming global coordination into local scheduling. Each node independently executes a Longest Processing Time First (LPT) spraying scheduler to proactively balance traffic using local information. RailS activates N parallel rails for fine-grained, topology-aware multipath transmission. Across synthetic and real-world MoE workloads, RailS improves bus bandwidth by 20%--78% and reduces completion time by 17%--78%. For Mixtral workloads, it shortens iteration time by 18%--40% and achieves near-optimal load balance, fully exploiting architectural parallelism in distributed training.
@article{arxiv.2510.19262,
title = {RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training},
author = {Heng Xu and Zhiwei Yu and Chengze Du and Ying Zhou and Letian Li and Haojie Wang and Weiqiang Cheng and Jialong Li},
journal= {arXiv preprint arXiv:2510.19262},
year = {2025}
}