Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
Abstract
Speculative decoding is a standard method for accelerating the inference speed of large language models. However, scaling it for production environments poses several engineering challenges, including efficiently implementing different operations (e.g., tree attention and multi-round speculative decoding) on GPU. In this paper, we detail the training and inference optimization techniques that we have implemented to enable EAGLE-based speculative decoding at a production scale for Llama models. With these changes, we achieve a new state-of-the-art inference latency for Llama models. For example, Llama4 Maverick decodes at a speed of about 4 ms per token (with a batch size of one) on 8 NVIDIA H100 GPUs, which is 10% faster than the previously best known method. Furthermore, for EAGLE-based speculative decoding, our optimizations enable us to achieve a speed-up for large batch sizes between 1.4x and 2.0x at production scale.
Cite
@article{arxiv.2508.08192,
title = {Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions},
author = {Bangsheng Tang and Carl Chengyan Fu and Fei Kou and Grigory Sizov and Haoci Zhang and Jason Park and Jiawen Liu and Jie You and Qirui Yang and Sachin Mehta and Shengyong Cai and Xiaodong Wang and Xingyu Liu and Yunlu Li and Yanjun Zhou and Wei Wei and Zhiwei Zhao and Zixi Qi and Adolfo Victoria and Aya Ibrahim and Bram Wasti and Changkyu Kim and Daniel Haziza and Fei Sun and Giancarlo Delfin and Emily Guo and Jialin Ouyang and Jaewon Lee and Jianyu Huang and Jeremy Reizenstein and Lu Fang and Quinn Zhu and Ria Verma and Vlad Mihailescu and Xingwen Guo and Yan Cui and Ye Hu and Yejin Lee},
journal= {arXiv preprint arXiv:2508.08192},
year = {2025}
}
Comments
15 pages