With the increasing intelligence and autonomy of LLM agents, their potential applications in the legal domain are becoming increasingly apparent. However, existing general-domain benchmarks cannot fully capture the complexity and subtle nuances of real-world judicial cognition and decision-making. Therefore, we propose LegalAgentBench, a comprehensive benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain. LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge. We designed a scalable task construction framework and carefully annotated 300 tasks. These tasks span various types, including multi-hop reasoning and writing, and range across different difficulty levels, effectively reflecting the complexity of real-world legal scenarios. Moreover, beyond evaluating final success, LegalAgentBench incorporates keyword analysis during intermediate processes to calculate progress rates, enabling more fine-grained evaluation. We evaluated eight popular LLMs, highlighting the strengths, limitations, and potential areas for improvement of existing models and methods. LegalAgentBench sets a new benchmark for the practical application of LLMs in the legal domain, with its code and data available at \url{https://github.com/CSHaitao/LegalAgentBench}.
@article{arxiv.2412.17259,
title = {LegalAgentBench: Evaluating LLM Agents in Legal Domain},
author = {Haitao Li and Junjie Chen and Jingli Yang and Qingyao Ai and Wei Jia and Youfeng Liu and Kai Lin and Yueyue Wu and Guozhi Yuan and Yiran Hu and Wuyue Wang and Yiqun Liu and Minlie Huang},
journal= {arXiv preprint arXiv:2412.17259},
year = {2024}
}