First, do NOHARM: towards clinically safe large language models
Abstract
Large language models (LLMs) are routinely used by physicians and patients for medical advice, yet their clinical safety profiles remain poorly characterized. We present NOHARM (Numerous Options Harm Assessment for Risk in Medicine), a benchmark using 100 real primary care-to-specialist consultation cases to measure frequency and severity of harm from LLM-generated medical recommendations. NOHARM covers 10 specialties, with 12,747 expert annotations for 4,249 clinical management options. Across 31 LLMs, potential for severe harm from LLM recommendations occurs in up to 22.2% (95% CI 21.6-22.8%) of cases, with harm of omission accounting for 76.6% (95% CI 76.4-76.8%) of errors. Safety performance is only moderately correlated (r = 0.61-0.64) with existing AI and medical knowledge benchmarks. The best models outperform generalist physicians on safety (mean difference 9.7%, 95% CI 7.0-12.5%), and a diverse multi-agent approach improves safety compared to solo models (mean difference 8.0%, 95% CI 4.0-12.1%). Therefore, despite strong performance on existing evaluations, widely used AI models can produce severely harmful medical advice at nontrivial rates, underscoring clinical safety as a distinct performance dimension necessitating explicit measurement.
Cite
@article{arxiv.2512.01241,
title = {First, do NOHARM: towards clinically safe large language models},
author = {David Wu and Fateme Nateghi Haredasht and Saloni Kumar Maharaj and Priyank Jain and Jessica Tran and Matthew Gwiazdon and Arjun Rustagi and Jenelle Jindal and Jacob M. Koshy and Vinay Kadiyala and Anup Agarwal and Bassman Tappuni and Brianna French and Sirus Jesudasen and Christopher V. Cosgriff and Rebanta Chakraborty and Jillian Caldwell and Susan Ziolkowski and David J. Iberri and Robert Diep and Rahul S. Dalal and Kira L. Newman and Kristin Galetta and J. Carl Pallais and Nancy Wei and Kathleen M. Buchheit and David I. Hong and Ernest Y. Lee and Allen Shih and Vartan Pahalyants and Tamara B. Kaplan and Vishnu Ravi and Sarita Khemani and April S. Liang and Daniel Shirvani and Advait Patil and Nicholas Marshall and Kanav Chopra and Joel Koh and Adi Badhwar and Liam G. McCoy and David J. H. Wu and Yingjie Weng and Sumant Ranji and Kevin Schulman and Nigam H. Shah and Jason Hom and Arnold Milstein and Adam Rodman and Jonathan H. Chen and Ethan Goh},
journal= {arXiv preprint arXiv:2512.01241},
year = {2025}
}