Large language models (LLMs) have revolutionized the state-of-the-art of many different natural language processing tasks. Although serving LLMs is computationally and memory demanding, the rise of Small Language Models (SLMs) offers new opportunities for resource-constrained users, who now are able to serve small models with cutting-edge performance. In this paper, we present a set of experiments designed to benchmark SLM inference at performance and energy levels. Our analysis provides a new perspective in serving, highlighting that the small memory footprint of SLMs allows for reaching the Pareto-optimal throughput within the resource capacity of a single accelerator. In this regard, we present an initial set of findings demonstrating how model replication can effectively improve resource utilization for serving SLMs.
@article{arxiv.2404.03353,
title = {Towards Pareto Optimal Throughput in Small Language Model Serving},
author = {Pol G. Recasens and Yue Zhu and Chen Wang and Eun Kyung Lee and Olivier Tardieu and Alaa Youssef and Jordi Torres and Josep Ll. Berral},
journal= {arXiv preprint arXiv:2404.03353},
year = {2025}
}
Comments
Revised version of the paper published at EuroMLSys'24, fix figure 6 and 7