Soohak：面向评估大语言模型研究级数学能力的数学家精选基准
计算与语言
2026-05-20 v3
摘要
在前沿大语言模型在 IMO 上取得金牌成绩之后，社区正在寻找下一 meaningful且具有挑战性的目标来衡量 LLM 推理能力。虽然奥林匹克式问题衡量的是逐步推理，但研究水平问题以这种推理去推动数学知识本身的前沿发展，因而成为引人入胜的替代方案。然而，研究水平数学基准仍稀缺，因为此类问题难以获取（如 Riemann Bench 和 FrontierMath-Tier 4 分别包含 25 和 50 个问题）。为支持可靠的下一代前沿模型评估，我们引入 Soohak，一个由 64 位数学家从零开始编写的 439 题基准。Soohak 包含两个子集。在挑战子集上，前沿模型包括 Gemini-3-Pro、GPT-5 和 Claude-Opus-4.5 分别达到 30.4%、26.4% 和 10.4%，仍有显著提升空间，而领先的开源权重模型如 Qwen3-235B、GPT-OSS-120B 和 Kimi-2.5 均低于 15%。值得注意的是，除了标准问题解决外，Soohak 引入了拒绝子集，用于探测研究数学中固有的能力：识别问题无解并暂停，而非生成自信但无依据的答案。在此子集上，没有模型超过 50%，将拒绝识别 identified as 新的优化目标，而当前模型未直接解决。为防止数据污染，数据集将于 2026 年底公开，模型评估在此期间可请求获得。
引用
@article{arxiv.2605.09063,
title = {Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs},
author = {Guijin Son and Seungone Kim and Catherine Arnett and Hyunwoo Ko and Hyein Lee and Hyeonah Kang and Jiang Longxi and Jin Yun and JungYup Lee and Kyungmin Lee and Sam Yoosuk Kim and Sang Park and Seunghyeok Hong and SeungJae Lee and Seungyeop Yi and Shinae Shin and SunHye Bok and Sunyoung Shin and Yonghoon Ji and Youngtaek Kim and Hanearl Jung and Akari Asai and Graham Neubig and Sean Welleck and Youngjae Yu and Akshelin R and Alexander B. Ivanov and Boboev Muhammadjon and Chae Young Han and Christian Stump and Cooper R. Anderson and Dmitrii Karp and Dohyun Kwon and Dongryung Yi and DoYong Kwon and Duk-Soon Oh and Eunho Choi and Giovanni Resta and Greta Panova and Huiyun Noh and Hyungryul Baik and Hyungsun Bae and Inomov Mashrafdzhon and Jeewon Kim and Jeong-Rae Kim and Ji Eun Lee and Jiaqi Liu and Jieui Kang and Jimin Kim and Jon-Lark Kim and Joonyeong Won and Junseo Yoon and Junwoo Jo and Kibeom Kim and Kiwoon Kwon and Mario Kummer and Max Mercer and Min Hoon Kim and Minjun Kim and Nahyun Lee and Ng Ze-An and Nicolas Libedinsky and Rafał Marcin Łochowski and Raphaël Lachièze-Rey and Robert Auffarth and Ruichen Zhang and Sejin Park and Seonguk Seo and Shin Jaehoon and Sunatullo and Taewoong Eom and Yeachan Park and Yongseok Jang and Youchan Oh and Zhaoyang Wang and Zoltán Kovács},
journal= {arXiv preprint arXiv:2605.09063},
year = {2026}
}
备注
Under review, For questions or model-evaluation requests, contact [email protected]$