使用Concordia评估LLM智能体在混合动机场景中的泛化能力
人工智能
2025-12-04 v1
摘要
大型语言模型（LLM）智能体在社交互动方面展示了令人印象深刻的能力，并且越来越多地被部署在可能与人类和人工智能体交互的场景中。这些交互代表了LLM智能体的关键前沿，然而现有的评估方法未能衡量这些能力如何泛化到新的社交情境。在本文中，我们引入了一种使用Concordia（一种自然语言多智能体模拟环境）评估LLM智能体在零样本混合动机环境中合作能力的方法。我们的方法通过测试智能体在多样化伙伴和情境中识别并利用互利机会的能力来衡量通用合作智能。我们展示了NeurIPS 2024 Concordia竞赛的经验结果，智能体在从谈判到集体行动问题的一系列多样化场景中评估其达成互利的能力。我们的发现揭示了当前智能体能力与可靠合作所需的鲁棒泛化之间存在显著差距，特别是在需要说服和规范执行的场景中。
引用
@article{arxiv.2512.03318,
title = {Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia},
author = {Chandler Smith and Marwa Abdulhai and Manfred Diaz and Marko Tesic and Rakshit S. Trivedi and Alexander Sasha Vezhnevets and Lewis Hammond and Jesse Clifton and Minsuk Chang and Edgar A. Duéñez-Guzmán and John P. Agapiou and Jayd Matyas and Danny Karmon and Akash Kundu and Aliaksei Korshuk and Ananya Ananya and Arrasy Rahman and Avinaash Anand Kulandaivel and Bain McHale and Beining Zhang and Buyantuev Alexander and Carlos Saith Rodriguez Rojas and Caroline Wang and Chetan Talele and Chenao Liu and Chichen Lin and Diana Riazi and Di Yang Shi and Emanuel Tewolde and Elizaveta Tennant and Fangwei Zhong and Fuyang Cui and Gang Zhao and Gema Parreño Piqueras and Hyeonggeun Yun and Ilya Makarov and Jiaxun Cui and Jebish Purbey and Jim Dilkes and Jord Nguyen and Lingyun Xiao and Luis Felipe Giraldo and Manuela Chacon-Chamorro and Manuel Sebastian Rios Beltran and Marta Emili García Segura and Mengmeng Wang and Mogtaba Alim and Nicanor Quijano and Nico Schiavone and Olivia Macmillan-Scott and Oswaldo Peña and Peter Stone and Ram Mohan Rao Kadiyala and Rolando Fernandez and Ruben Manrique and Sunjia Lu and Sheila A. McIlraith and Shamika Dhuri and Shuqing Shi and Siddhant Gupta and Sneheel Sarangi and Sriram Ganapathi Subramanian and Taehun Cha and Toryn Q. Klassen and Wenming Tu and Weijian Fan and Wu Ruiyang and Xue Feng and Yali Du and Yang Liu and Yiding Wang and Yipeng Kang and Yoonchang Sung and Yuxuan Chen and Zhaowei Zhang and Zhihan Wang and Zhiqiang Wu and Ziang Chen and Zilong Zheng and Zixia Jia and Ziyan Wang and Dylan Hadfield-Menell and Natasha Jaques and Tim Baarslag and Jose Hernandez-Orallo and Joel Z. Leibo},
journal= {arXiv preprint arXiv:2512.03318},
year = {2025}
}
备注
Published at NeurIPS Datasets and Benchmarks 2025, 10 pages