Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable. To remedy this issue, we develop the first differentially private (DP) retrieval-augmented generation algorithm that is capable of generating high-quality image samples while providing provable privacy guarantees. Specifically, we assume access to a text-to-image diffusion model trained on a small amount of public data, and design a DP retrieval mechanism to augment the text prompt with samples retrieved from a private retrieval dataset. Our \emph{differentially private retrieval-augmented diffusion model} (DP-RDM) requires no fine-tuning on the retrieval dataset to adapt to another domain, and can use state-of-the-art generative models to generate high-quality image samples while satisfying rigorous DP guarantees. For instance, when evaluated on MS-COCO, our DP-RDM can generate samples with a privacy budget of ϵ=10, while providing a 3.5 point improvement in FID compared to public-only retrieval for up to 10,000 queries.
@article{arxiv.2403.14421,
title = {DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning},
author = {Jonathan Lebensold and Maziar Sanjabi and Pietro Astolfi and Adriana Romero-Soriano and Kamalika Chaudhuri and Mike Rabbat and Chuan Guo},
journal= {arXiv preprint arXiv:2403.14421},
year = {2024}
}