In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: https://github.com/Raman1121/Diffusion_Memorization_HPO
@article{arxiv.2410.22149,
title = {Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models},
author = {Raman Dutt and Pedro Sanchez and Ondrej Bohdal and Sotirios A. Tsaftaris and Timothy Hospedales},
journal= {arXiv preprint arXiv:2410.22149},
year = {2024}
}
Comments
Accepted at the GenLaw (Generative AI + Law) workshop at ICML'24