Deploying Large Language Model (LLM) applications, particularly those relying on Retrieval-Augmented Generation (RAG), remains challenging due to high computational demands, outdated knowledge bases, and the need to manually select optimal pipeline components. In this work, we propose a modular framework for benchmarking and guiding the efficient development of RAG applications by focusing on resource telemetry and component recommendation, suggesting the best components for a domain-specific dataset. Our approach leverages core techniques in LLM applications, including document chunking, vector databases, embedding models, and retrievers, to evaluate trade-offs among accuracy, efficiency, and scalability. By directly correlating retrieval and generation quality with underlying hardware constraints, RAGe supports researchers to identify the most effective, domain-specific RAG setups for their specific operational needs, facilitating rapid prototyping even on consumer-grade hardware.
@article{arxiv.2605.27445,
title = {RAGe: A Retrieval-Augmented Generation Evaluation Framework},
author = {Larissa Guder and João Pedro de Moura and Arthur Accorsi and Gustavo Losch do Amaral and Maurício Cecílio Magnaguagno and Felipe Meneguzzi and Marcio Sorraglia Pinho and Dalvan Griebler},
journal= {arXiv preprint arXiv:2605.27445},
year = {2026}
}