Uncertainty in Fairness Assessment: Maintaining Stable Conclusions Despite Fluctuations
Abstract
Several recent works encourage the use of a Bayesian framework when assessing performance and fairness metrics of a classification algorithm in a supervised setting. We propose the Uncertainty Matters (UM) framework that generalizes a Beta-Binomial approach to derive the posterior distribution of any criteria combination, allowing stable performance assessment in a bias-aware setting.We suggest modeling the confusion matrix of each demographic group using a Multinomial distribution updated through a Bayesian procedure. We extend UM to be applicable under the popular K-fold cross-validation procedure. Experiments highlight the benefits of UM over classical evaluation frameworks regarding informativeness and stability.
Cite
@article{arxiv.2302.01079,
title = {Uncertainty in Fairness Assessment: Maintaining Stable Conclusions Despite Fluctuations},
author = {Ainhize Barrainkua and Paula Gordaliza and Jose A. Lozano and Novi Quadrianto},
journal= {arXiv preprint arXiv:2302.01079},
year = {2023}
}
Comments
25 pages (including references and appendix), 10 figures. Submitted to ICML 2023