English

Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple

Computation and Language 2026-03-13 v1 Information Theory Machine Learning math.IT

Abstract

Speculative decoding is a technique that uses multiple language models to accelerate infer- ence. Previous works have used an experi- mental approach to optimize the throughput of the inference pipeline, which involves LLM training and can be costly. This study of spec- ulative decoding proposes a theory that ana- lytically connects the key hyperparameters of pre-trained LLMs to the throughput efficiency of a downstream SD-based inference system. The theory allows the prediction of throughput- optimal hyperparameters for the components of an inference system before their pre-training.

Keywords

Cite

@article{arxiv.2603.11053,
  title  = {Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple},
  author = {Amirhossein Bozorgkhoo and Igor Molybog},
  journal= {arXiv preprint arXiv:2603.11053},
  year   = {2026}
}