Large language models (LLMs) have achieved success, but cost and privacy constraints necessitate deploying smaller models locally while offloading complex queries to cloud-based models. Existing router evaluations are unsystematic, overlooking scenario-specific requirements and out-of-distribution robustness. We propose RouterXBench, a principled evaluation framework with three dimensions: router ability, scenario alignment, and cross-domain robustness. Unlike prior work that relies on output probabilities or external embeddings, we utilize internal hidden states that capture model uncertainty before answer generation. We introduce ProbeDirichlet, a lightweight router that aggregates cross-layer hidden states via learnable Dirichlet distributions with probabilistic training. Trained on multi-domain data, it generalizes robustly across in-domain and out-of-distribution scenarios. Our results show ProbeDirichlet achieves 16.68% and 18.86% relative improvements over the best baselines in router ability and high-accuracy scenarios, with consistent performance across model families, model scales, heterogeneous tasks, and agentic workflows.
@article{arxiv.2602.11877,
title = {Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems},
author = {Wanxing Wu and He Zhu and Yixia Li and Lei Yang and Jiehui Zhao and Hongru Wang and Jian Yang and Benyou Wang and Bingyi Jing and Guanhua Chen},
journal= {arXiv preprint arXiv:2602.11877},
year = {2026}
}
Comments
Our code is publicly available at https://github.com/zhuchichi56/RouterXBench