Reducing Class-Wise Performance Disparity via Margin Regularization
Abstract
Deep neural networks often exhibit substantial disparities in class-wise accuracy, even when trained on class-balanced data, posing concerns for reliable deployment. While prior efforts have explored empirical remedies, a theoretical understanding of such performance disparities in classification remains limited. In this work, we present Margin Regularization for Performance Disparity Reduction (MR), a theoretically principled regularization for classification by dynamically adjusting margins in both the logit and representation spaces. Our analysis establishes a margin-based, class-sensitive generalization bound that reveals how per-class feature variability contributes to error, motivating the use of larger margins for hard classes. Guided by this insight, MR optimizes per-class logit margins proportional to feature spread and penalizes excessive representation margins to enhance intra-class compactness. Experiments on seven datasets, including ImageNet, and diverse pre-trained backbones (MAE, MoCov2, CLIP) demonstrate that MR not only improves overall accuracy but also significantly boosts hard class performance without trading off easy classes, thus reducing performance disparity. Code is available at: https://github.com/BeierZhu/MR2
Cite
@article{arxiv.2602.00205,
title = {Reducing Class-Wise Performance Disparity via Margin Regularization},
author = {Beier Zhu and Kesen Zhao and Jiequan Cui and Qianru Sun and Yuan Zhou and Xun Yang and Hanwang Zhang},
journal= {arXiv preprint arXiv:2602.00205},
year = {2026}
}
Comments
To appear in ICLR 2026