English

CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding

Artificial Intelligence 2026-05-05 v1

Abstract

Vision-language models (VLMs) have demonstrated strong capabilities in multimodal perception and reasoning. However, deploying large VLMs on mobile devices remains challenging due to their substantial computational and memory demands. A practical alternative is device-edge co-inference, where a lightweight draft VLM on the mobile device collaborates with a larger target VLM on the edge server via speculative decoding. Nevertheless, directly extending speculative decoding to VLMs suffers from severe inefficiency due to excessive visual-token computation and high communication overhead. To address these challenges, we propose CoVSpec, an efficient collaborative speculative decoding framework for VLM inference. Specifically, we first develop a training-free visual token reduction framework that prunes redundant visual tokens on the mobile device by jointly considering query relevance, token activity, and low-rank dependency. Moreover, we design an adaptive drafting strategy that dynamically adjusts both the verification frequency and the draft length. In addition, we introduce a parallel branching mechanism with decoupled verification-correction to improve draft-side utilization during target-side verification and reduce correction-related transmission overhead. Experiments on multiple benchmarks show that CoVSpec achieves up to 2.21x higher throughput than target-only inference and reduces communication overhead by more than 96% compared with baselines, without compromising task accuracy.

Keywords

Cite

@article{arxiv.2605.02218,
  title  = {CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding},
  author = {Yuanyuan Jia and Shunpu Tang and Qianqian Yang},
  journal= {arXiv preprint arXiv:2605.02218},
  year   = {2026}
}

Comments

6 pages, 2 tables, 1 figure. Submitted to IEEE Globecom 2026

R2 v1 2026-07-01T12:47:58.031Z