Why is the winner the best?
Abstract
International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.
Cite
@article{arxiv.2303.17719,
title = {Why is the winner the best?},
author = {Matthias Eisenmann and Annika Reinke and Vivienn Weru and Minu Dietlinde Tizabi and Fabian Isensee and Tim J. Adler and Sharib Ali and Vincent Andrearczyk and Marc Aubreville and Ujjwal Baid and Spyridon Bakas and Niranjan Balu and Sophia Bano and Jorge Bernal and Sebastian Bodenstedt and Alessandro Casella and Veronika Cheplygina and Marie Daum and Marleen de Bruijne and Adrien Depeursinge and Reuben Dorent and Jan Egger and David G. Ellis and Sandy Engelhardt and Melanie Ganz and Noha Ghatwary and Gabriel Girard and Patrick Godau and Anubha Gupta and Lasse Hansen and Kanako Harada and Mattias Heinrich and Nicholas Heller and Alessa Hering and Arnaud Huaulmé and Pierre Jannin and Ali Emre Kavur and Oldřich Kodym and Michal Kozubek and Jianning Li and Hongwei Li and Jun Ma and Carlos Martín-Isla and Bjoern Menze and Alison Noble and Valentin Oreiller and Nicolas Padoy and Sarthak Pati and Kelly Payette and Tim Rädsch and Jonathan Rafael-Patiño and Vivek Singh Bawa and Stefanie Speidel and Carole H. Sudre and Kimberlin van Wijnen and Martin Wagner and Donglai Wei and Amine Yamlahi and Moi Hoon Yap and Chun Yuan and Maximilian Zenk and Aneeq Zia and David Zimmerer and Dogu Baran Aydogan and Binod Bhattarai and Louise Bloch and Raphael Brüngel and Jihoon Cho and Chanyeol Choi and Qi Dou and Ivan Ezhov and Christoph M. Friedrich and Clifton Fuller and Rebati Raman Gaire and Adrian Galdran and Álvaro García Faura and Maria Grammatikopoulou and SeulGi Hong and Mostafa Jahanifar and Ikbeom Jang and Abdolrahim Kadkhodamohammadi and Inha Kang and Florian Kofler and Satoshi Kondo and Hugo Kuijf and Mingxing Li and Minh Huan Luu and Tomaž Martinčič and Pedro Morais and Mohamed A. Naser and Bruno Oliveira and David Owen and Subeen Pang and Jinah Park and Sung-Hong Park and Szymon Płotka and Elodie Puybareau and Nasir Rajpoot and Kanghyun Ryu and Numan Saeed and Adam Shephard and Pengcheng Shi and Dejan Štepec and Ronast Subedi and Guillaume Tochon and Helena R. Torres and Helene Urien and João L. Vilaça and Kareem Abdul Wahid and Haojie Wang and Jiacheng Wang and Liansheng Wang and Xiyue Wang and Benedikt Wiestler and Marek Wodzinski and Fangfang Xia and Juanying Xie and Zhiwei Xiong and Sen Yang and Yanwu Yang and Zixuan Zhao and Klaus Maier-Hein and Paul F. Jäger and Annette Kopp-Schneider and Lena Maier-Hein},
journal= {arXiv preprint arXiv:2303.17719},
year = {2023}
}
Comments
accepted to CVPR 2023