Differentially private stochastic gradient descent (DP-SGD) allows models to be trained in a privacy-preserving manner, but has proven difficult to scale to the era of foundation models. We introduce DP-ZO, a private fine-tuning framework for large language models by privatizing zeroth order optimization methods. A key insight into the design of our method is that the direction of the gradient in the zeroth-order optimization we use is random and the only information from training data is the step size, i.e., a scalar. Therefore, we only need to privatize the scalar step size, which is memory-efficient. DP-ZO provides a strong privacy-utility trade-off across different tasks, and model sizes that are comparable to DP-SGD in (ε,δ)-DP. Notably, DP-ZO possesses significant advantages over DP-SGD in memory efficiency, and obtains higher utility in ε-DP when using the Laplace mechanism.
@article{arxiv.2401.04343,
title = {Private Fine-tuning of Large Language Models with Zeroth-order Optimization},
author = {Xinyu Tang and Ashwinee Panda and Milad Nasr and Saeed Mahloujifar and Prateek Mittal},
journal= {arXiv preprint arXiv:2401.04343},
year = {2025}
}