QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
Abstract
As Large Language Models (LLMs) saturate elementary benchmarks, the research frontier has shifted from generation to the reliability of automated evaluation. We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics. To quantify this, we introduce QEDBench, the first large-scale dual-rubric alignment benchmark to systematically measure alignment with human experts on university-level math proofs by contrasting course-specific rubrics against expert common knowledge criteria. By deploying a dual-evaluation matrix (7 judges x 5 solvers) against 1,000+ hours of human evaluation, we reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias (up to +0.18, +0.20, +0.30, +0.36 mean score inflation, respectively). Furthermore, we uncover a critical reasoning gap in the discrete domain: while Gemini 3.0 Pro achieves state-of-the-art performance (0.91 average human evaluation score), other reasoning models like GPT-5 Pro and Claude Sonnet 4.5 see their performance significantly degrade in discrete domains. Specifically, their average human evaluation scores drop to 0.72 and 0.63 in Discrete Math, and to 0.74 and 0.50 in Graph Theory. In addition to these research results, we also release QEDBench as a public benchmark for evaluating and improving AI judges. Our benchmark is publicly published at https://github.com/qqliu/Yale-QEDBench.
Cite
@article{arxiv.2602.20629,
title = {QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs},
author = {Santiago Gonzalez and Alireza Amiri Bavandpour and Peter Ye and Edward Zhang and Ruslans Aleksejevs and Todor Antić and Polina Baron and Sujeet Bhalerao and Shubhrajit Bhattacharya and Zachary Burton and John Byrne and Hyungjun Choi and Nujhat Ahmed Disha and Koppany István Encz and Yuchen Fang and Robert Joseph George and Ebrahim Ghorbani and Alan Goldfarb and Jing Guo and Meghal Gupta and Stefano Huber and Annika Kanckos and Minjung Kang and Hyun Jong Kim and Dino Lorenzini and Levi Lorenzo and Tianyi Mao and Giovanni Marzenta and Ariane M. Masuda and Lukas Mauth and Ana Mickovic and Andres Miniguano-Trujillo and Antoine Moulin and Wenqi Ni and Tomos Parry and Kevin Ren and Hossein Roodbarani and Mathieu Rundström and Manjil Saikia and Detchat Samart and Rebecca Steiner and Connor Stewart and Dhara Thakkar and Jeffrey Tse and Vasiliki Velona and Yunhai Xiang and Sibel Yalçın and Jun Yan and Ji Zeng and Arman Cohan and Quanquan C. Liu},
journal= {arXiv preprint arXiv:2602.20629},
year = {2026}
}