NL2Repo-Bench：面向编程智能体长期仓库生成能力的评估
计算与语言
2026-01-09 v2
摘要
近期编程智能体的进展表明自主软件开发正快速推进，但现有基准测试未能严格评估构建完整软件系统所需的长期能力。此前大多数评估聚焦于局部代码生成、脚手架补全或短期修复任务，留下了关于智能体能否在真实仓库构建所要求的扩展时间范围内保持一致推理、规划和执行的问题。为填补这一空白，我们提出 NL2Repo Bench，一个专门用于评估编程智能体长期仓库生成能力的基准测试。仅给定一份自然语言需求文档和一个空工作区，智能体必须自主设计架构、管理依赖、实现多模块逻辑，并生成一个可完整安装的 Python 库。我们对最先进的开源和闭源模型进行的实验表明，长期仓库生成在很大程度上仍未解决：即使最强的智能体平均测试通过率也低于 40%，且很少能完整完成整个仓库。详细分析揭示了根本性的长期失败模式，包括过早终止、全局一致性丧失、脆弱的跨文件依赖以及数百个交互步骤中的规划不足。NL2Repo Bench 建立了一个严格的、可验证的测试平台，用于衡量持续智能体能力，并指出长期推理是下一代自主编程智能体的核心瓶颈。
引用
@article{arxiv.2512.12730,
title = {NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents},
author = {Jingzhe Ding and Shengda Long and Changxin Pu and Huan Zhou and Hongwan Gao and Xiang Gao and Chao He and Yue Hou and Fei Hu and Zhaojian Li and Weiran Shi and Zaiyuan Wang and Daoguang Zan and Chenchen Zhang and Xiaoxu Zhang and Qizhi Chen and Xianfu Cheng and Bo Deng and Qingshui Gu and Kai Hua and Juntao Lin and Pai Liu and Mingchen Li and Xuanguang Pan and Zifan Peng and Yujia Qin and Yong Shan and Zhewen Tan and Weihao Xie and Zihan Wang and Yishuo Yuan and Jiayu Zhang and Enduo Zhao and Yunfei Zhao and He Zhu and Liya Zhu and Chenyang Zou and Ming Ding and Jianpeng Jiao and Jiaheng Liu and Minghao Liu and Qian Liu and Chongyang Tao and Jian Yang and Tong Yang and Zhaoxiang Zhang and Xinjie Chen and Wenhao Huang and Ge Zhang},
journal= {arXiv preprint arXiv:2512.12730},
year = {2026}
}