Agent-Testing Agent:用于自动化测试和评估的元智能体
摘要
LLM agents 越来越多地被部署 to plan, retrieve, and write with tools,yet evaluation still leans on static benchmarks and small human studies。 We present the Agent-Testing Agent(ATA),a meta-agent that combines static code analysis, designer interrogation, literature mining, and persona-driven adversarial test generation whose difficulty adapts via judge feedback。 Each dialogue is scored with an LLM-as-a-Judge(LAAJ) rubric and used to steer subsequent tests toward the agent's weakest capabilities。 On a travel planner and a Wikipedia writer, the ATA surfaces more diverse and severe failures than expert annotators while matching severity, and finishes in 20--30 minutes versus ten-annotator rounds that took days。 Ablating code analysis and web search increases variance and miscalibration, underscoring the value of evidence-grounded test generation。 The ATA outputs quantitative metrics and qualitative bug reports for developers。 We release the full methodology and open-source implementation for reproducible agent testing: https://github.com/KhalilMrini/Agent-Testing-Agent。
引用
@article{arxiv.2508.17393,
title = {Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents},
author = {Sameer Komoravolu and Khalil Mrini},
journal= {arXiv preprint arXiv:2508.17393},
year = {2025}
}