每一次FLOP都至关重要：在非高端GPU上扩展300B混合专家LING大语言模型
机器学习
2025-03-11 v2 人工智能
计算与语言
摘要
在本技术报告中，我们应对训练大规模混合专家（MoE）模型的挑战，重点关注克服此类系统中普遍存在的成本低效和资源限制问题。为解决这些问题，我们提出了两个不同规模的MoE大语言模型（LLM），即Ling-Lite和Ling-Plus（中文称为“百灵”，拼音为Bǎilíng）。Ling-Lite包含168亿参数，激活参数为27.5亿，而Ling-Plus拥有2900亿参数，激活参数为288亿。两个模型都展现出与领先行业基准相当的性能。本报告提供了可操作的见解，以提高资源受限环境中AI开发的效率和可及性，促进更具可扩展性和可持续性的技术。具体而言，为降低大规模MoE模型的训练成本，我们提出了创新方法，用于（1）优化模型架构和训练流程，（2）改进训练异常处理，以及（3）提升模型评估效率。此外，利用从知识图谱生成的高质量数据，我们的模型在工具使用方面展现出优于其他模型的能力。最终，我们的实验结果表明，一个300B的MoE LLM可以在性能较低的设备上有效训练，同时达到与类似规模模型（包括稠密和MoE模型）相当的性能。与高性能设备相比，在预训练阶段使用规格较低的硬件系统可显著节省成本，计算成本降低约20%。模型可在 https://huggingface.co/inclusionAI 获取。
引用
@article{arxiv.2503.05139,
title = {Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs},
author = {Ling Team and Binwei Zeng and Chao Huang and Chao Zhang and Changxin Tian and Cong Chen and Dingnan Jin and Feng Yu and Feng Zhu and Feng Yuan and Fakang Wang and Gangshan Wang and Guangyao Zhai and Haitao Zhang and Huizhong Li and Jun Zhou and Jia Liu and Junpeng Fang and Junjie Ou and Jun Hu and Ji Luo and Ji Zhang and Jian Liu and Jian Sha and Jianxue Qian and Jiewei Wu and Junping Zhao and Jianguo Li and Jubao Feng and Jingchao Di and Junming Xu and Jinghua Yao and Kuan Xu and Kewei Du and Longfei Li and Lei Liang and Lu Yu and Li Tang and Lin Ju and Peng Xu and Qing Cui and Song Liu and Shicheng Li and Shun Song and Song Yan and Tengwei Cai and Tianyi Chen and Ting Guo and Ting Huang and Tao Feng and Tao Wu and Wei Wu and Xiaolu Zhang and Xueming Yang and Xin Zhao and Xiaobo Hu and Xin Lin and Yao Zhao and Yilong Wang and Yongzhen Guo and Yuanyuan Wang and Yue Yang and Yang Cao and Yuhao Fu and Yi Xiong and Yanzhe Li and Zhe Li and Zhiqiang Zhang and Ziqi Liu and Zhaoxin Huan and Zujie Wen and Zhenhang Sun and Zhuoxuan Du and Zhengyu He},
journal= {arXiv preprint arXiv:2503.05139},
year = {2025}
}
备注
34 pages