Pangu Ultra MoE：如何在昇腾NPU上训练你的大型MoE模型
计算与语言
2025-05-08 v1
摘要
具有混合专家（MoE）架构且参数规模接近万亿的稀疏大语言模型（LLMs）正主导着最强语言模型的领域。然而，庞大的模型规模对底层的软件和硬件系统提出了重大挑战。在本文中，我们旨在揭示在昇腾NPU上驾驭如此规模模型的秘诀。关键目标是更好地利用动态稀疏模型结构下的计算资源，并在实际硬件上实现预期的性能增益。为了在不重复进行昂贵实验的情况下选择适合昇腾NPU的模型配置，我们利用模拟来比较各种模型超参数的权衡。这项研究催生了Pangu Ultra MoE，一个拥有7180亿参数的稀疏LLM，我们在该模型上进行了实验以验证模拟结果。在系统方面，我们深入研究了专家并行，以优化NPU设备间的通信，从而减少同步开销。我们还优化了设备内的内存效率，以进一步降低参数和激活值管理的开销。最终，在6K昇腾NPU上训练Pangu Ultra MoE时，我们实现了30.0%的模型浮点运算利用率（MFU），性能与DeepSeek R1相当，并证明了昇腾系统能够驾驭最先进语言模型的所有训练阶段。大量实验表明，我们的方法可以实现带有MoE的大规模稀疏语言模型的高效训练。我们还研究了此类模型的行为，以供未来参考。
引用
@article{arxiv.2505.04519,
title = {Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs},
author = {Yehui Tang and Yichun Yin and Yaoyuan Wang and Hang Zhou and Yu Pan and Wei Guo and Ziyang Zhang and Miao Rang and Fangcheng Liu and Naifu Zhang and Binghan Li and Yonghan Dong and Xiaojun Meng and Yasheng Wang and Dong Li and Yin Li and Dandan Tu and Can Chen and Youliang Yan and Fisher Yu and Ruiming Tang and Yunhe Wang and Botian Huang and Bo Wang and Boxiao Liu and Changzheng Zhang and Da Kuang and Fei Liu and Gang Huang and Jiansheng Wei and Jiarui Qin and Jie Ran and Jinpeng Li and Jun Zhao and Liang Dai and Lin Li and Liqun Deng and Peifeng Qin and Pengyuan Zeng and Qiang Gu and Shaohua Tang and Shengjun Cheng and Tao Gao and Tao Yu and Tianshu Li and Tianyu Bi and Wei He and Weikai Mao and Wenyong Huang and Wulong Liu and Xiabing Li and Xianzhi Yu and Xueyu Wu and Xu He and Yangkai Du and Yan Xu and Ye Tian and Yimeng Wu and Yongbing Huang and Yong Tian and Yong Zhu and Yue Li and Yufei Wang and Yuhang Gai and Yujun Li and Yu Luo and Yunsheng Ni and Yusen Sun and Zelin Chen and Zhe Liu and Zhicheng Liu and Zhipeng Tu and Zilin Ding and Zongyuan Zhan},
journal= {arXiv preprint arXiv:2505.04519},
year = {2025}
}