We observe that the traditional use of DP with the Adam optimizer introduces a bias in the second moment estimation, due to the addition of independent noise in the gradient computation. This bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam, and Adam's sign descent interpretation. Empirically, correcting the bias introduced by DP noise significantly improves the optimization performance of DP-Adam.
@article{arxiv.2304.11208,
title = {DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation},
author = {Qiaoyue Tang and Mathias Lécuyer},
journal= {arXiv preprint arXiv:2304.11208},
year = {2023}
}
Comments
Published at ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models