P/D-Serve: Serving Disaggregated Large Language Model at Scale
Abstract
Serving disaggregated large language models (LLMs) over tens of thousands of xPU devices (GPUs or NPUs) with reliable performance faces multiple challenges. 1) Ignoring the diversity (various prefixes and tidal requests), treating all the prompts in a mixed pool is inadequate. To facilitate the similarity per scenario and minimize the inner mismatch on P/D (prefill and decoding) processing, fine-grained organization is required, dynamically adjusting P/D ratios for better performance. 2) Due to inaccurate estimation on workload (queue status or maintained connections), the global scheduler easily incurs unnecessary timeouts in prefill. 3) Block-fixed device-to-device (D2D) KVCache transfer over cluster-level RDMA (remote direct memory access) fails to achieve desired D2D utilization as expected. To overcome previous problems, this paper proposes an end-to-end system P/D-Serve, complying with the paradigm of MLOps (machine learning operations), which models end-to-end (E2E) P/D performance and enables: 1) fine-grained P/D organization, mapping the service with RoCE (RDMA over converged ethernet) as needed, to facilitate similar processing and dynamic adjustments on P/D ratios; 2) on-demand forwarding upon rejections for idle prefill, decoupling the scheduler from regular inaccurate reports and local queues, to avoid timeouts in prefill; and 3) efficient KVCache transfer via optimized D2D access. P/D-Serve is implemented upon Ascend and MindSpore, has been deployed over tens of thousands of NPUs for more than eight months in commercial use, and further achieves 60\%, 42\% and 46\% improvements on E2E throughput, time-to-first-token (TTFT) SLO (service level objective) and D2D transfer time. As the E2E system with optimizations, P/D-Serve achieves 6.7x increase on throughput, compared with aggregated LLMs.
Cite
@article{arxiv.2408.08147,
title = {P/D-Serve: Serving Disaggregated Large Language Model at Scale},
author = {Yibo Jin and Tao Wang and Huimin Lin and Mingyang Song and Peiyang Li and Yipeng Ma and Yicheng Shan and Zhengfan Yuan and Cailong Li and Yajing Sun and Tiandeng Wu and Xing Chu and Ruizhi Huan and Li Ma and Xiao You and Wenting Zhou and Yunpeng Ye and Wen Liu and Xiangkun Xu and Yongsheng Zhang and Tiantian Dong and Jiawei Zhu and Zhe Wang and Xijian Ju and Jianxun Song and Haoliang Cheng and Xiaojing Li and Jiandong Ding and Hefei Guo and Zhengyong Zhang},
journal= {arXiv preprint arXiv:2408.08147},
year = {2024}
}