我们需要多少标注者？——观察者间变异性对自动有丝分裂象评估可靠性影响的研究
计算机视觉与模式识别
2022-03-17 v2
摘要
组织学切片中有丝分裂象的密度是许多肿瘤预后相关的特征。由于病理学家间的高变异性，基于深度学习的算法是改善肿瘤预后的一个有前景的方案。病理学家是数据库开发的金标准，然而标注错误可能阻碍准确算法的开发。在本工作中，我们评估了多专家共识（n = 3、5、7、9、11）对算法性能的益处。虽然使用个体数据库训练导致高度可变的 F 分数，但使用三位标注者的共识时性能显著提升且更为一致。增加更多标注者仅带来微小改进。我们得出结论：由少数病理学家构建且标签准确性高的数据库可能是高算法性能与时间投入之间的最佳折衷。
引用
@article{arxiv.2012.02495,
title = {How Many Annotators Do We Need? -- A Study on the Influence of Inter-Observer Variability on the Reliability of Automatic Mitotic Figure Assessment},
author = {Frauke Wilm and Christof A. Bertram and Christian Marzahl and Alexander Bartel and Taryn A. Donovan and Charles-Antoine Assenmacher and Kathrin Becker and Mark Bennett and Sarah Corner and Brieuc Cossic and Daniela Denk and Martina Dettwiler and Beatriz Garcia Gonzalez and Corinne Gurtner and Annika Lehmbecker and Sophie Merz and Stephanie Plog and Anja Schmidt and Rebecca C. Smedley and Marco Tecilla and Tuddow Thaiwong and Katharina Breininger and Matti Kiupel and Andreas Maier and Robert Klopfleisch and Marc Aubreville},
journal= {arXiv preprint arXiv:2012.02495},
year = {2022}
}
备注
Due to data inconsistencies experiments had to be repeated with a reduced number of annotators (17 in version 1). All findings of the previous version were reproducible. 7 pages, 2 figures, accepted at BVM workshop 2021