Medical Imaging AI Competitions Lack Fairness
Abstract
Benchmarking competitions are central to the development of artificial intelligence (AI) in medical imaging, defining performance standards and shaping methodological progress. However, it remains unclear whether these benchmarks provide data that are sufficiently representative, accessible, and reusable to support clinically meaningful AI. In this work, we assess fairness along two complementary dimensions: (1) whether challenge datasets are representative of real-world clinical diversity, and (2) whether they are accessible and legally reusable in line with the FAIR principles. To address this question, we conducted a large-scale systematic study of 241 biomedical image analysis challenges comprising 458 tasks across 19 imaging modalities. Our findings show substantial biases in dataset composition, including geographic location, modality-, and problem type-related biases, indicating that current benchmarks do not adequately reflect real-world clinical diversity. Despite their widespread influence, challenge datasets were frequently constrained by restrictive or ambiguous access conditions, inconsistent or non-compliant licensing practices, and incomplete documentation, limiting reproducibility and long-term reuse. Together, these shortcomings expose foundational fairness limitations in our benchmarking ecosystem and highlight a disconnect between leaderboard success and clinical relevance.
Cite
@article{arxiv.2512.17581,
title = {Medical Imaging AI Competitions Lack Fairness},
author = {Annika Reinke and Evangelia Christodoulou and Sthuthi Sadananda and A. Emre Kavur and Khrystyna Faryna and Daan Schouten and Bennett A. Landman and Carole Sudre and Olivier Colliot and Nick Heller and Sophie Loizillon and Martin Maška and Maëlys Solal and Arya Yazdan-Panah and Vilma Bozgo and Ömer Sümer and Siem de Jong and Sophie Fischer and Michal Kozubek and Tim Rädsch and Nadim Hammoud and Fruzsina Molnár-Gábor and Steven Hicks and Michael A. Riegler and Anindo Saha and Vajira Thambawita and Pal Halvorsen and Amelia Jiménez-Sánchez and Qingyang Yang and Veronika Cheplygina and Sabrina Bottazzi and Alexander Seitel and Spyridon Bakas and Alexandros Karargyris and Kiran Vaidhya Venkadesh and Bram van Ginneken and Lena Maier-Hein},
journal= {arXiv preprint arXiv:2512.17581},
year = {2025}
}
Comments
Submitted to Nature BME