Metrics reloaded: Recommendations for image analysis validation
Abstract
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases.
Cite
@article{arxiv.2206.01653,
title = {Metrics reloaded: Recommendations for image analysis validation},
author = {Lena Maier-Hein and Annika Reinke and Patrick Godau and Minu D. Tizabi and Florian Buettner and Evangelia Christodoulou and Ben Glocker and Fabian Isensee and Jens Kleesiek and Michal Kozubek and Mauricio Reyes and Michael A. Riegler and Manuel Wiesenfarth and A. Emre Kavur and Carole H. Sudre and Michael Baumgartner and Matthias Eisenmann and Doreen Heckmann-Nötzel and Tim Rädsch and Laura Acion and Michela Antonelli and Tal Arbel and Spyridon Bakas and Arriel Benis and Matthew Blaschko and M. Jorge Cardoso and Veronika Cheplygina and Beth A. Cimini and Gary S. Collins and Keyvan Farahani and Luciana Ferrer and Adrian Galdran and Bram van Ginneken and Robert Haase and Daniel A. Hashimoto and Michael M. Hoffman and Merel Huisman and Pierre Jannin and Charles E. Kahn and Dagmar Kainmueller and Bernhard Kainz and Alexandros Karargyris and Alan Karthikesalingam and Hannes Kenngott and Florian Kofler and Annette Kopp-Schneider and Anna Kreshuk and Tahsin Kurc and Bennett A. Landman and Geert Litjens and Amin Madani and Klaus Maier-Hein and Anne L. Martel and Peter Mattson and Erik Meijering and Bjoern Menze and Karel G. M. Moons and Henning Müller and Brennan Nichyporuk and Felix Nickel and Jens Petersen and Nasir Rajpoot and Nicola Rieke and Julio Saez-Rodriguez and Clara I. Sánchez and Shravya Shetty and Maarten van Smeden and Ronald M. Summers and Abdel A. Taha and Aleksei Tiulpin and Sotirios A. Tsaftaris and Ben Van Calster and Gaël Varoquaux and Paul F. Jäger},
journal= {arXiv preprint arXiv:2206.01653},
year = {2024}
}
Comments
Shared first authors: Lena Maier-Hein, Annika Reinke. arXiv admin note: substantial text overlap with arXiv:2104.05642 Published in Nature Methods