MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
摘要
Accurate evaluation of conversational retrieval is pivotal for advancing Retrieval-Augmented Generation (RAG) systems. However, existing conversational retrieval benchmarks suffer from costly, sparse human annotation or rigid, unnatural automated heuristics. To address these challenges, we introduce MTR-Suite, a unified framework for auditing, synthesizing, and benchmarking retrieval. It features: (1) MTR-Eval, an LLM-based auditor quantifying alignment gaps in previous benchmarks; (2) MTR-Pipeline, a multi-agent system using greedy traversal clustering to generate high-fidelity dialogues at 1/400th human cost; and (3) MTR-Bench, a rigorous general-domain benchmark. MTR-Bench mimics production-style challenges (hard topic switching, verbosity), offering superior discriminative power. We make our code and data publicly available to facilitate future research at https://github.com/rangehow/mtr-suite.
引用
@article{arxiv.2605.20729,
title = {MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks},
author = {Junhao Ruan and Abudukeyumu Abudula and Bei Li and Yongjing Yin and Xinyu Liu and Kechen Jiao and Xin Chen and Jingang Wang and Xunliang Cai and Tong Xiao and Jingbo Zhu},
journal= {arXiv preprint arXiv:2605.20729},
year = {2026}
}
备注
Accepted to ACL 2026 (main conference). 28 pages. Code and data: https://github.com/rangehow/mtr-suite