Instability in clinical risk stratification models using deep learning
Abstract
While it has been well known in the ML community that deep learning models suffer from instability, the consequences for healthcare deployments are under characterised. We study the stability of different model architectures trained on electronic health records, using a set of outpatient prediction tasks as a case study. We show that repeated training runs of the same deep learning model on the same training data can result in significantly different outcomes at a patient level even though global performance metrics remain stable. We propose two stability metrics for measuring the effect of randomness of model training, as well as mitigation strategies for improving model stability.
Cite
@article{arxiv.2211.10828,
title = {Instability in clinical risk stratification models using deep learning},
author = {Daniel Lopez-Martinez and Alex Yakubovich and Martin Seneviratne and Adam D. Lelkes and Akshit Tyagi and Jonas Kemp and Ethan Steinberg and N. Lance Downing and Ron C. Li and Keith E. Morse and Nigam H. Shah and Ming-Jun Chen},
journal= {arXiv preprint arXiv:2211.10828},
year = {2022}
}
Comments
Accepted for publication in Machine Learning for Health (ML4H) 2022