English

Scalable Training of Mixture-of-Experts Models with Megatron Core

Distributed, Parallel, and Cluster Computing 2026-03-11 v2 Computation and Language Machine Learning

Abstract

Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation, creating coupled constraints across memory, communication, and computation. Optimizing one dimension often shifts pressure to another, demanding co-design across the full system stack. We address these challenges for MoE training through integrated optimizations spanning memory (fine-grained recomputation, offloading, etc.), communication (optimized dispatchers, overlapping, etc.), and computation (Grouped GEMM, fusions, CUDA Graphs, etc.). The framework also provides Parallel Folding for flexible multi-dimensional parallelism, low-precision training support for FP8 and NVFP4, and efficient long-context training. On NVIDIA GB300 and GB200, it achieves 1,233/1,048 TFLOPS/GPU for DeepSeek-V3-685B and 974/919 TFLOPS/GPU for Qwen3-235B. As a performant, scalable, and production-ready open-source solution, it has been used across academia and industry for training MoE models ranging from billions to trillions of parameters on clusters scaling up to thousands of GPUs. This report explains how these techniques work, their trade-offs, and their interactions at the systems level, providing practical guidance for scaling MoE models with Megatron Core.

Keywords

Cite

@article{arxiv.2603.07685,
  title  = {Scalable Training of Mixture-of-Experts Models with Megatron Core},
  author = {Zijie Yan and Hongxiao Bai and Xin Yao and Dennis Liu and Tong Liu and Hongbin Liu and Pingtian Li and Evan Wu and Shiqing Fan and Li Tao and Robin Zhang and Yuzhong Wang and Shifang Xu and Jack Chang and Xuwen Chen and Kunlun Li and Yan Bai and Gao Deng and Nan Zheng and Vijay Anand Korthikanti and Abhinav Khattar and Ethan He and Soham Govande and Sangkug Lym and Zhongbo Zhu and Qi Zhang and Haochen Yuan and Xiaowei Ren and Deyu Fu and Tailai Ma and Shunkang Zhang and Jiang Shao and Ray Wang and Vasudevan Rengasamy and Rachit Garg and Santosh Bhavani and Xipeng Li and Chandler Zhou and David Wu and Yingcan Wei and Ashwath Aithal and Michael Andersch and Mohammad Shoeybi and Jiajie Yao and June Yang},
  journal= {arXiv preprint arXiv:2603.07685},
  year   = {2026}
}

Comments

Technical Report. 88 pages. 42 figures