The rapid advancement of large language models (LLMs) has led to the rise of LLM-based agents. Recent research shows that multi-agent systems (MAS), where each agent plays a specific role, can outperform individual LLMs. However, configuring an MAS for a task remains challenging, with performance only observable post-execution. Inspired by scaling laws in LLM development, we investigate whether MAS performance can be predicted beforehand. We introduce AgentMonitor, a framework that integrates at the agent level to capture inputs and outputs, transforming them into statistics for training a regression model to predict task performance. Additionally, it can further apply real-time corrections to address security risks posed by malicious agents, mitigating negative impacts and enhancing MAS security. Experiments demonstrate that an XGBoost model achieves a Spearman correlation of 0.89 in-domain and 0.58 in more challenging scenarios. Furthermore, using AgentMonitor reduces harmful content by 6.2% and increases helpful content by 1.8% on average, enhancing safety and reliability. Code is available at \url{https://github.com/chanchimin/AgentMonitor}.
@article{arxiv.2408.14972,
title = {AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems},
author = {Chi-Min Chan and Jianxuan Yu and Weize Chen and Chunyang Jiang and Xinyu Liu and Weijie Shi and Zhiyuan Liu and Wei Xue and Yike Guo},
journal= {arXiv preprint arXiv:2408.14972},
year = {2024}
}