Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pattern repeatedly processes the same prompt prefix across models. Consequently, each model redundantly executes the prefill stage and maintains its own key-value (KV) cache, increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt. To address this issue, we propose PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple models in a disaggregated setting. PrefillShare factorizes the model into prefill and decode modules, freezes the prefill module, and fine-tunes only the decode module. This design allows multiple task-specific models to share a prefill module and the KV cache generated for the same prompt. We further introduce a routing mechanism that enables effective prefill sharing across heterogeneous models in a vLLM-based disaggregated system. PrefillShare not only matches full fine-tuning accuracy on a broad range of tasks and models, but also delivers 4.5x lower p95 latency and 3.9x higher throughput in multi-model agent workloads.
@article{arxiv.2602.12029,
title = {PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving},
author = {Sunghyeon Woo and Hoseung Kim and Sunghwan Shim and Minjung Jo and Hyunjoon Jeong and Jeongtae Lee and Joonghoon Kim and Sungjae Lee and Baeseong Park and Se Jung Kwon and Dongsoo Lee},
journal= {arXiv preprint arXiv:2602.12029},
year = {2026}
}