Mobile and IoT applications increasingly adopt deep learning inference to provide intelligence. Inference requests are typically sent to a cloud infrastructure over a wireless network that is highly variable, leading to the challenge of dynamic Service Level Objectives (SLOs) at the request level. This paper presents Sponge, a novel deep learning inference serving system that maximizes resource efficiency while guaranteeing dynamic SLOs. Sponge achieves its goal by applying in-place vertical scaling, dynamic batching, and request reordering. Specifically, we introduce an Integer Programming formulation to capture the resource allocation problem, providing a mathematical model of the relationship between latency, batch size, and resources. We demonstrate the potential of Sponge through a prototype implementation and preliminary experiments and discuss future works.
@article{arxiv.2404.00704,
title = {Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling},
author = {Kamran Razavi and Saeid Ghafouri and Max Mühlhäuser and Pooyan Jamshidi and Lin Wang},
journal= {arXiv preprint arXiv:2404.00704},
year = {2024}
}