Large Language Models drive a wide range of modern AI applications but impose substantial challenges on large-scale serving systems due to intensive computation, strict latency constraints, and throughput bottlenecks. We introduce OmniInfer, a unified system-level acceleration framework designed to maximize end-to-end serving efficiency through fine-grained optimization of expert placement, cache compression, and scheduling. OmniInfer integrates three complementary components: OmniPlacement for load-aware Mixture-of-Experts scheduling, OmniAttn for sparse attention acceleration, and OmniProxy for disaggregation-aware request scheduling. Built atop vLLM, OmniInfer delivers system-wide performance gains through adaptive resource disaggregation, efficient sparsity exploitation, and global coordination across prefill and decode phases. Evaluated on DeepSeek-R1 within a 10-node Ascend 910C cluster, OmniInfer achieves 616 QPM, where the unified framework reduces TPOT by 36\%, and the superimposition of OmniProxy further slashes TTFT by 38\%. The project is open-sourced at [this https URL](https://gitee.com/omniai/omniinfer).
@article{arxiv.2511.22481,
title = {OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency},
author = {Jun Wang and Yunxiang Yao and Wenwei Kuang and Runze Mao and Zhenhao Sun and Zhuang Tao and Ziyang Zhang and Dengyu Li and Jiajun Chen and Zhili Wang and Kai Cui and Congzhi Cai and Longwen Lan and Ken Zhang},
journal= {arXiv preprint arXiv:2511.22481},
year = {2025}
}