English

Predictive Prefetching for Retrieval-Augmented Generation

Computation and Language 2026-05-19 v1 Artificial Intelligence

Abstract

Retrieval-Augmented Generation (RAG) improves factual grounding in large language models but suffers from substantial latency due to synchronous retrieval. While recent work explores asynchronous retrieval, existing approaches rely on heuristic coordination between retrieval and generation and assume stable information demands during decoding that often break in complex, multi-domain settings. In this paper, we propose an advanced asynchronous retrieval framework that enables predictive prefetching aligned with evolving information needs. The framework explicitly predicts when retrieval should be triggered and what information should be retrieved using three components, a retrieval predictor, a context monitor, and a query generator, by exploiting semantic precursors in generation dynamics that emerge several tokens before uncertainty becomes critical. Experiments on multiple benchmarks demonstrate up to 43.5% end-to-end latency reduction and 62.4% improvement in time-to-first-token, while maintaining answer quality comparable to synchronous RAG baselines.

Keywords

Cite

@article{arxiv.2605.17989,
  title  = {Predictive Prefetching for Retrieval-Augmented Generation},
  author = {Wuyang Zhang and Shichao Pei},
  journal= {arXiv preprint arXiv:2605.17989},
  year   = {2026}
}

Comments

Accepted by Forty-third International Conference on Machine Learning ICML 2026