Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
Abstract
Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challenging target for measuring LLM reasoning. Whereas olympiad-style problems measure step-by-step reasoning alone, research-level problems use such reasoning to advance the frontier of mathematical knowledge itself, emerging as a compelling alternative. Yet research-level math benchmarks remain scarce because such problems are difficult to source (e.g., Riemann Bench and FrontierMath-Tier 4 contain 25 and 50 problems, respectively). To support reliable evaluation of next-generation frontier models, we introduce Soohak, a 439-problem benchmark newly authored from scratch by 64 mathematicians. Soohak comprises two subsets. On the Challenge subset, frontier models including Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 reach 30.4%, 26.4%, and 10.4% respectively, leaving substantial headroom, while leading open-weight models such as Qwen3-235B, GPT-OSS-120B, and Kimi-2.5 remain below 15%. Notably, beyond standard problem solving, Soohak introduces a refusal subset that probes a capability intrinsic to research mathematics: recognizing ill-posed problems and pausing rather than producing confident but unjustified answers. On this subset, no model exceeds 50%, identifying refusal as a new optimization target that current models do not directly address. To prevent contamination, the dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.
Cite
@article{arxiv.2605.09063,
title = {Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs},
author = {Guijin Son and Seungone Kim and Catherine Arnett and Hyunwoo Ko and Hyein Lee and Hyeonah Kang and Jiang Longxi and Jin Yun and JungYup Lee and Kyungmin Lee and Sam Yoosuk Kim and Sang Park and Seunghyeok Hong and SeungJae Lee and Seungyeop Yi and Shinae Shin and SunHye Bok and Sunyoung Shin and Yonghoon Ji and Youngtaek Kim and Hanearl Jung and Akari Asai and Graham Neubig and Sean Welleck and Youngjae Yu and Akshelin R and Alexander B. Ivanov and Boboev Muhammadjon and Chae Young Han and Christian Stump and Cooper R. Anderson and Dmitrii Karp and Dohyun Kwon and Dongryung Yi and DoYong Kwon and Duk-Soon Oh and Eunho Choi and Giovanni Resta and Greta Panova and Huiyun Noh and Hyungryul Baik and Hyungsun Bae and Inomov Mashrafdzhon and Jeewon Kim and Jeong-Rae Kim and Ji Eun Lee and Jiaqi Liu and Jieui Kang and Jimin Kim and Jon-Lark Kim and Joonyeong Won and Junseo Yoon and Junwoo Jo and Kibeom Kim and Kiwoon Kwon and Mario Kummer and Max Mercer and Min Hoon Kim and Minjun Kim and Nahyun Lee and Ng Ze-An and Nicolas Libedinsky and Rafał Marcin Łochowski and Raphaël Lachièze-Rey and Robert Auffarth and Ruichen Zhang and Sejin Park and Seonguk Seo and Shin Jaehoon and Sunatullo and Taewoong Eom and Yeachan Park and Yongseok Jang and Youchan Oh and Zhaoyang Wang and Zoltán Kovács},
journal= {arXiv preprint arXiv:2605.09063},
year = {2026}
}
Comments
Under review, For questions or model-evaluation requests, contact [email protected]$