Speculative decoding can significantly accelerate LLM inference, especially given that its cloud-edge collaborative deployment offers cloud workload offloading, offline robustness, and privacy enhancement. However, existing collaborative inference frameworks with speculative decoding are constrained by (i) sequential token generation and communication with low resource utilization, and (ii) inflexible cloud non-autoregressive verification (NAV) triggering that induces premature verification or costly rollbacks. In this paper, we propose PipeSD, an efficient cloud-edge collaborative pipeline inference framework with speculative decoding. PipeSD overlaps token generation and communication by a token-batch pipeline scheduling mechanism optimized by dynamic programming, and improves verification flexibility through a dual-threshold NAV triggering mechanism with a lightweight Bayesian optimization autotuner. We implement PipeSD using llama-cpp-python, PyTorch, and FastAPI, and evaluate it on a real-world cloud-edge testbed with two draft-target model pairs across four scenarios. Results show that PipeSD consistently outperforms state-of-the-art baselines, achieving 1.16x-2.16x speedup and reducing energy consumption by 14.3%-25.3%.
@article{arxiv.2605.13319,
title = {PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding},
author = {Yunhe Han and Yunqi Gao and Bing Hu and Mahdi Boloursaz Mashhadi and Yitong Duan and Pei Xiao and Yanfeng Zhang},
journal= {arXiv preprint arXiv:2605.13319},
year = {2026}
}
Comments
Accepted by ICML 2026. *Equal contribution. Yunhe Han and Yunqi Gao contributed equally to this work