When doctors are trained to diagnose a specific disease, they learn faster when presented with cases in order of increasing difficulty. This creates the need for automatically estimating how difficult it is for doctors to classify a given case. In this paper, we introduce methods for estimating how hard it is for a doctor to diagnose a case represented by a medical image, both when ground truth difficulties are available for training, and when they are not. Our methods are based on embeddings obtained with deep metric learning. Additionally, we introduce a practical method for obtaining ground truth human difficulty for each image case in a dataset using self-assessed certainty. We apply our methods to two different medical datasets, achieving high Kendall rank correlation coefficients, showing that we outperform existing methods by a large margin on our problem and data.
@article{arxiv.2203.11824,
title = {Was that so hard? Estimating human classification difficulty},
author = {Morten Rieger Hannemose and Josefine Vilsbøll Sundgaard and Niels Kvorning Ternov and Rasmus R. Paulsen and Anders Nymark Christensen},
journal= {arXiv preprint arXiv:2203.11824},
year = {2022}
}