Multi-modal Large Language Models (MLLMs) serving systems commonly employ KV-cache compression to reduce memory footprint. However, existing compression methods introduce significant processing overhead and queuing delays, particularly in concurrent serving scenarios. We present \texttt{FastCache}, a novel serving framework that effectively addresses these challenges through two key innovations: (1) a dynamic batching strategy that optimizes request scheduling across prefill, compression, and decode stages, and (2) an efficient KV-cache memory pool mechanism that eliminates memory fragmentation while maintaining high GPU utilization. Our comprehensive experiments on the GQA and MileBench datasets demonstrate that \texttt{FastCache} achieves up to 19.3× reduction in Time-To-First-Token (TTFT) and 12.1× improvement in throughput compared to state-of-the-art baselines. The system maintains stable performance under high-concurrency scenarios (up to 40 req/s) while reducing average memory consumption by 20\%. These results establish \texttt{FastCache} as an efficient solution for real-world LLM serving systems with KV-cache compression.
@article{arxiv.2503.08461,
title = {FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework},
author = {Jianian Zhu and Hang Wu and Haojie Wang and Yinghui Li and Biao Hou and Ruixuan Li and Jidong Zhai},
journal= {arXiv preprint arXiv:2503.08461},
year = {2025}
}