Large language models (LLMs) achieve remarkable performance across domains but remain prone to hallucinations and inconsistencies. Retrieval-augmented generation (RAG) mitigates these issues by augmenting model inputs with relevant documents retrieved from external sources. In many real-world scenarios, relevant knowledge is fragmented across organizations or institutions, motivating the need for federated search mechanisms that can aggregate results from heterogeneous data sources without centralizing the data. We introduce RAGRoute, a lightweight routing mechanism for federated search in RAG systems that dynamically selects relevant data sources at query time using a neural classifier, avoiding indiscriminate querying. This selective routing reduces communication overhead and end-to-end latency while preserving retrieval quality, achieving up to 80.65% reductions in communication volume and 52.50% reductions in latency across three benchmarks, while matching the accuracy of querying all sources.
@article{arxiv.2502.19280,
title = {Efficient Federated Search for Retrieval-Augmented Generation using Lightweight Routing},
author = {Akash Dhasade and Rachid Guerraoui and Anne-Marie Kermarrec and Diana Petrescu and Rafael Pires and Mathis Randl and Martijn de Vos},
journal= {arXiv preprint arXiv:2502.19280},
year = {2026}
}
Comments
To appear in the proceedings of DAIS 2026 (Distributed Applications and Interoperable Systems). An earlier version appeared at EuroMLSys 2025