PHYBench：大语言模型物理感知与推理的全面评估
计算与语言
2025-05-20 v2
摘要
当前用于评估大语言模型 (LLM) 推理能力的基准存在显著局限：任务过于简化、数据受污染以及评估项目存在缺陷。这些缺陷 necessitates 更严格的评估方法。为此，我们引入 PHYBench，这是一个包含 500 题原创物理问题的基准，涵盖高中阶段至物理奥林匹克难度。PHYBench 通过原创内容消除数据污染，并采用系统性的精选流程消除存在缺陷的评估项目。评估表明，PHYBench 激活更多 token，较其他基准（如 AIME 2024、OlympiadBench 和 GPQA）提供了更强的模型间差异。即使是表现最佳的模型 Gemini 2.5 Pro，准确率也仅为人类专家的 36.9%。为进一步提升评估精度，我们引入了表达式编辑距离 (EED) 评分方法，用于数学表达式评估，该方法比二元评分提高了 204% 的样本效率。此外，PHYBench 有效激发了多步骤和多条件推理，为检验模型推理鲁棒性、偏好和缺陷提供了平台。基准结果和数据集已公开于 https://www.phybench.cn/。
引用
@article{arxiv.2504.16074,
title = {PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models},
author = {Shi Qiu and Shaoyang Guo and Zhuo-Yang Song and Yunbo Sun and Zeyu Cai and Jiashen Wei and Tianyu Luo and Yixuan Yin and Haoxu Zhang and Yi Hu and Chenyang Wang and Chencheng Tang and Haoling Chang and Qi Liu and Ziheng Zhou and Tianyu Zhang and Jingtian Zhang and Zhangyi Liu and Minghao Li and Yuku Zhang and Boxuan Jing and Xianqi Yin and Yutong Ren and Zizhuo Fu and Jiaming Ji and Weike Wang and Xudong Tian and Anqi Lv and Laifu Man and Jianxiang Li and Feiyu Tao and Qihua Sun and Zhou Liang and Yushu Mu and Zhongxuan Li and Jing-Jun Zhang and Shutao Zhang and Xiaotian Li and Xingqi Xia and Jiawei Lin and Zheyu Shen and Jiahang Chen and Qiuhao Xiong and Binran Wang and Fengyuan Wang and Ziyang Ni and Bohan Zhang and Fan Cui and Changkun Shao and Qing-Hong Cao and Ming-xing Luo and Yaodong Yang and Muhan Zhang and Hua Xing Zhu},
journal= {arXiv preprint arXiv:2504.16074},
year = {2025}
}
备注
34 pages ,12 figures, 7 tables, latest update in 2025/05/18