Models driven by spurious correlations often yield poor generalization performance. We propose the counterfactual (CF) alignment method to detect and quantify spurious correlations of black box classifiers. Our methodology is based on counterfactual images generated with respect to one classifier being input into other classifiers to see if they also induce changes in the outputs of these classifiers. The relationship between these responses can be quantified and used to identify specific instances where a spurious correlation exists. This is validated by observing intuitive trends in face-attribute and waterbird classifiers, as well as by fabricating spurious correlations and detecting their presence, both visually and quantitatively. Furthermore, utilizing the CF alignment method, we demonstrate that we can evaluate robust optimization methods (GroupDRO, JTT, and FLAC) by detecting a reduction in spurious correlations.
@article{arxiv.2312.02186,
title = {Identifying Spurious Correlations using Counterfactual Alignment},
author = {Joseph Paul Cohen and Louis Blankemeier and Akshay Chaudhari},
journal= {arXiv preprint arXiv:2312.02186},
year = {2025}
}
Comments
Accepted to Transactions on Machine Learning Research (TMLR), Code: https://github.com/ieee8023/latentshift