Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
Abstract
We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field of LLM has been witnessing unprecedented advances in pushing the scale and capability of LLM in recent years, training such a large-scale model still involves significant optimization and system challenges. To stabilize the training process, we propose depth-scaled sandwich normalization, which effectively eliminates loss spikes during the training process of deep models. We pre-train our model on 13.2 trillion diverse and high-quality tokens and further enhance its reasoning capabilities during post-training. To perform such large-scale training efficiently, we utilize 8,192 Ascend NPUs with a series of system optimizations. Evaluations on multiple diverse benchmarks indicate that Pangu Ultra significantly advances the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse model structure contains much more parameters. Our exploration demonstrates that Ascend NPUs are capable of efficiently and effectively training dense models with more than 100 billion parameters. Our model and system will be available for our commercial customers.
Keywords
Cite
@article{arxiv.2504.07866,
title = {Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs},
author = {Yichun Yin and Wenyong Huang and Kaikai Song and Yehui Tang and Xueyu Wu and Wei Guo and Peng Guo and Yaoyuan Wang and Xiaojun Meng and Yasheng Wang and Dong Li and Can Chen and Dandan Tu and Yin Li and Fisher Yu and Ruiming Tang and Yunhe Wang and Baojun Wang and Bin Wang and Bo Wang and Boxiao Liu and Changzheng Zhang and Duyu Tang and Fei Mi and Hui Jin and Jiansheng Wei and Jiarui Qin and Jinpeng Li and Jun Zhao and Liqun Deng and Lin Li and Minghui Xu and Naifu Zhang and Nianzu Zheng and Qiang Li and Rongju Ruan and Shengjun Cheng and Tianyu Guo and Wei He and Wei Li and Weiwen Liu and Wulong Liu and Xinyi Dai and Yonghan Dong and Yu Pan and Yue Li and Yufei Wang and Yujun Li and Yunsheng Ni and Zhe Liu and Zhenhe Zhang and Zhicheng Liu},
journal= {arXiv preprint arXiv:2504.07866},
year = {2025}
}
Comments
fix conflicts of latex pacakges