Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift
Machine Learning
2023-12-14 v1 Artificial Intelligence
Abstract
Knowing if a model will generalize to data 'in the wild' is crucial for safe deployment. To this end, we study model disagreement notions that consider the full predictive distribution - specifically disagreement based on Hellinger distance, Jensen-Shannon and Kullback-Leibler divergence. We find that divergence-based scores provide better test error estimates and detection rates on out-of-distribution data compared to their top-1 counterparts. Experiments involve standard vision and foundation models.
Cite
@article{arxiv.2312.08033,
title = {Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift},
author = {Mona Schirmer and Dan Zhang and Eric Nalisnick},
journal= {arXiv preprint arXiv:2312.08033},
year = {2023}
}
Comments
Workshop on Distribution Shifts, 37th Conference on Neural Information Processing Systems (NeurIPS 2023)