One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selection
Abstract
The proliferation of large language models (LLMs) with varying computational costs and performance profiles presents a critical challenge for scalable, cost-effective deployment in real-world applications. We introduce a unified routing framework that leverages a single-head cross-attention mechanism to jointly model query and model embeddings, enabling dynamic selection of the optimal LLM for each input query. Our approach is evaluated on RouterBench, a large-scale, publicly available benchmark encompassing diverse LLM pools and domains. By explicitly capturing fine-grained query-model interactions, our router predicts both response quality and generation cost, achieving up to 6.6% improvement in Average Improvement in Quality (AIQ) and 2.9% in maximum performance over existing routers. To robustly balance performance and cost, we propose an exponential reward function that enhances stability across user preferences. The resulting architecture is lightweight, generalizes effectively across domains, and demonstrates improved efficiency compared to prior methods, establishing a new standard for cost-aware LLM routing.
Cite
@article{arxiv.2509.09782,
title = {One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selection},
author = {Roshini Pulishetty and Mani Kishan Ghantasala and Keerthy Kaushik Dasoju and Niti Mangwani and Vishal Garimella and Aditya Mate and Somya Chatterjee and Yue Kang and Ehi Nosakhare and Sadid Hasan and Soundar Srinivasan},
journal= {arXiv preprint arXiv:2509.09782},
year = {2025}
}