CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
Abstract
API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.
Cite
@article{arxiv.2501.05255,
title = {CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing},
author = {Yewei Song and Xunzhu Tang and Cedric Lothritz and Saad Ezzini and Jacques Klein and Tegawendé F. Bissyandé and Andrey Boytsov and Ulrick Ble and Anne Goujon},
journal= {arXiv preprint arXiv:2501.05255},
year = {2025}
}