This paper investigates the impact of domain-specific model fine-tuning and of reasoning mechanisms on the performance of question-answering (Q&A) systems powered by large language models (LLMs) and Retrieval-Augmented Generation (RAG). Using the FinanceBench SEC financial filings dataset, we observe that, for RAG, combining a fine-tuned embedding model with a fine-tuned LLM achieves better accuracy than generic models, with relatively greater gains attributable to fine-tuned embedding models. Additionally, employing reasoning iterations on top of RAG delivers an even bigger jump in performance, enabling the Q&A systems to get closer to human-expert quality. We discuss the implications of such findings, propose a structured technical design space capturing major technical components of Q&A AI, and provide recommendations for making high-impact technical choices for such components. We plan to follow up on this work with actionable guides for AI teams and further investigations into the impact of domain-specific augmentation in RAG and into agentic AI capabilities such as advanced planning and reasoning.
@article{arxiv.2404.11792,
title = {Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Study},
author = {Zooey Nguyen and Anthony Annunziata and Vinh Luong and Sang Dinh and Quynh Le and Anh Hai Ha and Chanh Le and Hong An Phan and Shruti Raghavan and Christopher Nguyen},
journal= {arXiv preprint arXiv:2404.11792},
year = {2024}
}
Comments
Fixed typo of OODA's score on harder-question set in Table 2