TeleChat3-MoE训练报告
计算与语言
2026-01-01 v1
摘要
TeleChat3-MoE是TeleChat大语言模型的最新系列，采用混合专家架构，参数规模从1050亿到超过一万亿，在昇腾NPU集群上端到端训练。本技术报告主要介绍底层训练基础设施，该设施能够可靠且高效地扩展到前沿模型规模。我们详细阐述了用于算子级和端到端数值精度验证的系统方法，确保跨硬件平台和分布式并行策略的一致性。此外，我们引入了一套性能优化方案，包括交错流水线调度、面向长序列训练的注意力感知数据调度、用于专家并行的分层与重叠通信，以及基于DVM的算子融合。我们还提出了一种系统化的并行框架，利用分析估计和整数线性规划来优化多维并行配置。此外，我们提出了针对集群级优化的方法论，以解决大规模训练任务中的主机端和设备端瓶颈。这些基础设施的进步在包含数千个设备的集群上实现了显著的吞吐量提升和近线性扩展，为在硬件生态系统上进行大规模语言模型开发提供了坚实的基础。
引用
@article{arxiv.2512.24157,
title = {Training Report of TeleChat3-MoE},
author = {Xinzhang Liu and Chao Wang and Zhihao Yang and Zhuo Jiang and Xuncheng Zhao and Haoran Wang and Lei Li and Dongdong He and Luobin Liu and Kaizhe Yuan and Han Gao and Zihan Wang and Yitong Yao and Sishi Xiong and Wenmin Deng and Haowei He and Kaidong Yu and Yu Zhao and Ruiyu Fang and Yuhao Jiang and Yingyan Li and Xiaohui Hu and Xi Yu and Jingqi Li and Yanwei Liu and Qingli Li and Xinyu Shi and Junhao Niu and Chengnuo Huang and Yao Xiao and Ruiwen Wang and Fengkai Li and Luwen Pu and Kaipeng Jia and Fubei Yao and Yuyao Huang and Xuewei He and Zhuoru Jiang and Ruiting Song and Rui Xue and Qiyi Xie and Jie Zhang and Zilu Huang and Zhaoxi Zhang and Zhilong Lu and Yanhan Zhang and Yin Zhang and Yanlei Xue and Zhu Yuan and Teng Su and Xin Jiang and Shuangyong Song and Yongxiang Li and Xuelong Li},
journal= {arXiv preprint arXiv:2512.24157},
year = {2026}
}