Why rankings of biomedical image analysis competitions should be interpreted with care
Abstract
International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
Keywords
Cite
@article{arxiv.1806.02051,
title = {Why rankings of biomedical image analysis competitions should be interpreted with care},
author = {Lena Maier-Hein and Matthias Eisenmann and Annika Reinke and Sinan Onogur and Marko Stankovic and Patrick Scholz and Tal Arbel and Hrvoje Bogunovic and Andrew P. Bradley and Aaron Carass and Carolin Feldmann and Alejandro F. Frangi and Peter M. Full and Bram van Ginneken and Allan Hanbury and Katrin Honauer and Michal Kozubek and Bennett A. Landman and Keno März and Oskar Maier and Klaus Maier-Hein and Bjoern H. Menze and Henning Müller and Peter F. Neher and Wiro Niessen and Nasir Rajpoot and Gregory C. Sharp and Korsuk Sirinukunwattana and Stefanie Speidel and Christian Stock and Danail Stoyanov and Abdel Aziz Taha and Fons van der Sommen and Ching-Wei Wang and Marc-André Weber and Guoyan Zheng and Pierre Jannin and Annette Kopp-Schneider},
journal= {arXiv preprint arXiv:1806.02051},
year = {2019}
}
Comments
Article published in Nature Communications: https://rdcu.be/bRmNr