With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lower bound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models' cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard-v2 we empirically validate CARROT's performance against several alternative routers.
Cite
@article{arxiv.2502.03261,
title = {CARROT: A Cost Aware Rate Optimal Router},
author = {Seamus Somerstep and Felipe Maia Polo and Allysson Flavio Melo de Oliveira and Prattyush Mangal and Mírian Silva and Onkar Bhardwaj and Mikhail Yurochkin and Subha Maity},
journal= {arXiv preprint arXiv:2502.03261},
year = {2025}
}