Adaptive Set-Mass Calibration with Conformal Prediction
Abstract
Reliable probabilities are critical in high-risk applications, yet common calibration criteria (confidence, class-wise) are only necessary for full distributional calibration, and post-hoc methods often lack distribution-free guarantees. We propose a set-based notion of calibration, cumulative mass calibration, and a corresponding empirical error measure: the Cumulative Mass Calibration Error (CMCE). We develop a new calibration procedure that starts with conformal prediction to obtain a set of labels that gives the desired coverage. We then instantiate two simple post-hoc calibrators: a mass normalization and a temperature scaling-based rule, tuned to the conformal constraint. On multi-class image benchmarks, especially with a large number of classes, our methods consistently improve CMCE and standard metrics (ECE, cw-ECE, MCE) over baselines, delivering a practical, scalable framework with theoretical guarantees.
Cite
@article{arxiv.2505.15437,
title = {Adaptive Set-Mass Calibration with Conformal Prediction},
author = {Daniil Kazantsev and Mohsen Guizani and Eric Moulines and Maxim Panov and Nikita Kotelevskii},
journal= {arXiv preprint arXiv:2505.15437},
year = {2025}
}