When the Oracle Misleads: Modeling the Consequences of Using Observable Rather than Potential Outcomes in Risk Assessment Instruments
Abstract
Risk Assessment Instruments (RAIs) are widely used to forecast adverse outcomes in domains such as healthcare and criminal justice. RAIs are commonly trained on observational data and are optimized to predict observable outcomes rather than potential outcomes, which are the outcomes that would occur absent a particular intervention. Examples of relevant potential outcomes include whether a patient's condition would worsen without treatment or whether a defendant would recidivate if released pretrial. We illustrate how RAIs which are trained to predict observable outcomes can lead to worse decision making, causing precisely the types of harm they are intended to prevent. This can occur even when the predictors are Bayes-optimal and there is no unmeasured confounding.
Keywords
Cite
@article{arxiv.2104.01921,
title = {When the Oracle Misleads: Modeling the Consequences of Using Observable Rather than Potential Outcomes in Risk Assessment Instruments},
author = {Alan Mishler and Niccolò Dalmasso},
journal= {arXiv preprint arXiv:2104.01921},
year = {2021}
}
Comments
6 pages, 3 figures. Presented at the workshop "'Do the right thing': machine learning and causal inference for improved decision making," NeurIPS 2019