Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPod
Abstract
Scaled-out MoE LLMs and scaled-up SuperPods create new systems challenges for production Model-as-a-Service (MaaS), requiring disaggregation, low-latency communication, and decentralized serving. This report presents xDeepServe, the production serving system behind Huawei Cloud's MaaS offering on CloudMatrix384, a 48-server SuperPod with 384 Ascend 910C chips connected by a high-bandwidth UB fabric and global shared memory. It serves models including DeepSeek, Kimi, GLM, Qwen, and MiniMax, among others. xDeepServe is built around Transformerless, a disaggregated execution architecture that decomposes transformer inference into modular units -- attention, feedforward, and MoE -- and supports disaggregated Prefill-Decode and MoE-Attention deployments. To enable disaggregation, we develop XCCL, a memory-semantic communication layer providing microsecond-level point-to-point and scalable all-to-all primitives, and we extend FlowServe with decentralized DP groups and techniques to mitigate stragglers and synchronization variance. In a peak decoding configuration, xDeepServe reaches 2400 tokens/s per Ascend 910C chip at ~50ms time-per-output-token (TPOT).
Cite
@article{arxiv.2508.02520,
title = {Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPod},
author = {Ao Xiao and Bangzheng He and Baoquan Zhang and Baoxing Huai and Bingji Wang and Bo Wang and Bo Xu and Boyi Hou and Chan Yang and Changhong Liu and Cheng Cui and Chenyu Zhu and Cong Feng and Daohui Wang and Dayun Lin and Duo Zhao and Fengshao Zou and Fu Wang and Gangqiang Zhang and Gengyuan Dan and Guanjie Chen and Guodong Guan and Guodong Yang and Haifeng Li and Haipei Zhu and Haley Li and Hao Feng and Hao Huang and Hao Xu and Hengrui Ma and Hengtao Fan and Hui Liu and Jia Li and Jiang Liu and Jiang Xu and Jie Meng and Jinhan Xin and Junhao Hu and Juwei Chen and Lan Yu and Lanxin Miao and Liang Liu and Linan Jing and Lu Zhou and Meina Han and Mingkun Deng and Mingyu Deng and Naitian Deng and Nizhong Lin and Peihan Zhao and Peng Pan and Pengfei Shen and Ping Li and Qi Zhang and Qian Wang and Qin ZhC Qingrong Xia and Qingyi Zhang and Qunchao Fu and Ren Guo and Ruimin Gao and Shaochun Li and Sheng Long and Shentian Li and Shining Wan and Shuai Shen and Shuangfu Zeng and Shuming Jing and Siqi Yang and Song Zhang and Tao Xu and Tianlin Du and Ting Chen and Wanxu Wu and Wei Jiang and Weinan Tong and Weiwei Chen and Wen Peng and Wenli Zhou and Wenquan Yang and Wenxin Liang and Xiang Liu and Xiaoli Zhou and Xin Jin and Xinyu Duan and Xu Li and Xu Zhang and Xusheng Chen and Yalong Shan and Yang Gan and Yao Lu and Yi Deng and Yi Zheng and Ying Xiong and Yingfei Zheng and Yiyun Zheng and Yizhou Shan and Yong Gao and Yong Zhang and Yongqiang Yang and Yuanjin Gong and Yue Yu and Yuetao Chen and Yukun Zhu and Yulong He and Yusu Zhao and Yuyan Wu and Zenan Zhang and Zhaojin Zhuo and Zhaoyang Ji and Zhefeng Wang and Zheng Wang and Zhenan Fan and Zhenhua Yang and Zhenli Sheng and Zhibin Yu and Zhigang Ji and Zhihao Ren and Zhipeng Bian and Zhixia Liu and Zhiyu Dong and Zhonghua Li and Zhou Yu and Zhuoming Shen and Zhuwei Peng and Zi Ye and Zihao Xiang and Zimin Fu and Zixuan Zhang},
journal= {arXiv preprint arXiv:2508.02520},
year = {2026}
}