User experience is a critical factor Large Language Model (LLM) serving systems must consider, where service level objectives (SLOs) considering the experience of individual requests and system level metrics (SLMs) considering the overall system performance are two key performance measures. However, we observe two notable issues in existing metrics: 1) manually delaying the delivery of some tokens can improve SLOs, and 2) actively abandoning requests that do not meet SLOs can improve SLMs, both of which are counterintuitive. In this paper, we revisit SLOs and SLMs in LLM serving, and propose a new SLO that aligns with user experience. Based on the SLO, we propose a comprehensive metric framework called smooth goodput, which integrates SLOs and SLMs to reflect the nature of user experience in LLM serving. Through this unified framework, we reassess the performance of different LLM serving systems under multiple workloads. Evaluation results show that our metric framework provides a more comprehensive view of token delivery and request processing, and effectively captures the optimal point of user experience and system performance with different serving strategies.
@article{arxiv.2410.14257,
title = {Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving},
author = {Zhibin Wang and Shipeng Li and Yuhang Zhou and Xue Li and Zhonghui Zhang and Nguyen Cam-Tu and Rong Gu and Chen Tian and Guihai Chen and Sheng Zhong},
journal= {arXiv preprint arXiv:2410.14257},
year = {2025}
}