This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production environment. By conditioning draft predictions on both context vectors and sampled tokens, we can train our speculators to efficiently predict high-quality n-grams, which the base model then accepts or rejects. This allows us to effectively predict multiple tokens per inference forward pass, accelerating wall-clock inference speeds of highly optimized base model implementations by a factor of 2-3x. We explore these initial results and describe next steps for further improvements.
@article{arxiv.2404.19124,
title = {Accelerating Production LLMs with Combined Token/Embedding Speculators},
author = {Davis Wertheimer and Joshua Rosenkranz and Thomas Parnell and Sahil Suneja and Pavithra Ranganathan and Raghu Ganti and Mudhakar Srivatsa},
journal= {arXiv preprint arXiv:2404.19124},
year = {2024}
}
Comments
Original upload 4/29/24, updated 6/6/24 with additional references to concurrent work