SuperGPQA：跨285个研究生学科扩展LLM评估
计算与语言
2025-03-31 v4
摘要
大型语言模型（LLMs）在数学、物理和计算机科学等主流学科已显示出惊人的专业能力。然而，人类知识涵盖超过200个专业学科，远超现有基准测试的范围。LLMs在许多专业领域（特别是轻型工业、农业和服务导向领域）的能力评估不充分。为此，我们提出SuperGPQA，一个全面评估285个学科中研究生水平知识和推理能力的基准。我们的基准采用新型的人类-LLM协作过滤机制，通过迭代精炼基于LLM响应和专家反馈，以消除平凡或模糊问题。我们的实验结果揭示了当前最先进LLMs在各类知识领域（例如，以推理为导向的模型DeepSeek-R1在SuperGPQA上取得最高准确率61.82%）仍有很大的提升空间，凸显了当前模型能力与人工通用智能之间巨大的差距。此外，我们就大规模标注过程的管理提供了全面见解，涉及超过80名专家标注员以及交互式的人类-LLM协作系统，为规模相当的未来研究 initiative 提供了有价值的实践指导。
引用
@article{arxiv.2502.14739,
title = {SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines},
author = {P Team and Xinrun Du and Yifan Yao and Kaijing Ma and Bingli Wang and Tianyu Zheng and King Zhu and Minghao Liu and Yiming Liang and Xiaolong Jin and Zhenlin Wei and Chujie Zheng and Kaixin Deng and Shawn Gavin and Shian Jia and Sichao Jiang and Yiyan Liao and Rui Li and Qinrui Li and Sirun Li and Yizhi Li and Yunwen Li and David Ma and Yuansheng Ni and Haoran Que and Qiyao Wang and Zhoufutu Wen and Siwei Wu and Tyshawn Hsing and Ming Xu and Zhenzhu Yang and Zekun Moore Wang and Junting Zhou and Yuelin Bai and Xingyuan Bu and Chenglin Cai and Liang Chen and Yifan Chen and Chengtuo Cheng and Tianhao Cheng and Keyi Ding and Siming Huang and Yun Huang and Yaoru Li and Yizhe Li and Zhaoqun Li and Tianhao Liang and Chengdong Lin and Hongquan Lin and Yinghao Ma and Tianyang Pang and Zhongyuan Peng and Zifan Peng and Qige Qi and Shi Qiu and Xingwei Qu and Shanghaoran Quan and Yizhou Tan and Zili Wang and Chenqing Wang and Hao Wang and Yiya Wang and Yubo Wang and Jiajun Xu and Kexin Yang and Ruibin Yuan and Yuanhao Yue and Tianyang Zhan and Chun Zhang and Jinyang Zhang and Xiyue Zhang and Xingjian Zhang and Yue Zhang and Yongchi Zhao and Xiangyu Zheng and Chenghua Zhong and Yang Gao and Zhoujun Li and Dayiheng Liu and Qian Liu and Tianyu Liu and Shiwen Ni and Junran Peng and Yujia Qin and Wenbo Su and Guoyin Wang and Shi Wang and Jian Yang and Min Yang and Meng Cao and Xiang Yue and Zhaoxiang Zhang and Wangchunshu Zhou and Jiaheng Liu and Qunshu Lin and Wenhao Huang and Ge Zhang},
journal= {arXiv preprint arXiv:2502.14739},
year = {2025}
}