Measuring what Matters: Construct Validity in Large Language Model Benchmarks
Abstract
Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as 'safety' and 'robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.
Cite
@article{arxiv.2511.04703,
title = {Measuring what Matters: Construct Validity in Large Language Model Benchmarks},
author = {Andrew M. Bean and Ryan Othniel Kearns and Angelika Romanou and Franziska Sofia Hafner and Harry Mayne and Jan Batzner and Negar Foroutan and Chris Schmitz and Karolina Korgul and Hunar Batra and Oishi Deb and Emma Beharry and Cornelius Emde and Thomas Foster and Anna Gausen and María Grandury and Simeng Han and Valentin Hofmann and Lujain Ibrahim and Hazel Kim and Hannah Rose Kirk and Fangru Lin and Gabrielle Kaili-May Liu and Lennart Luettgau and Jabez Magomere and Jonathan Rystrøm and Anna Sotnikova and Yushi Yang and Yilun Zhao and Adel Bibi and Antoine Bosselut and Ronald Clark and Arman Cohan and Jakob Foerster and Yarin Gal and Scott A. Hale and Inioluwa Deborah Raji and Christopher Summerfield and Philip H. S. Torr and Cozmin Ududec and Luc Rocher and Adam Mahdi},
journal= {arXiv preprint arXiv:2511.04703},
year = {2025}
}
Comments
39th Conference on Neural Information Processing Systems (NeurIPS 2025) Track on Datasets and Benchmarks