A Noise is Worth Diffusion Guidance
Abstract
Diffusion models excel in generating high-quality images. However, current diffusion models struggle to produce reliable images without guidance methods, such as classifier-free guidance (CFG). Are guidance methods truly necessary? Observing that noise obtained via diffusion inversion can reconstruct high-quality images without guidance, we focus on the initial noise of the denoising pipeline. By mapping Gaussian noise to `guidance-free noise', we uncover that small low-magnitude low-frequency components significantly enhance the denoising process, removing the need for guidance and thus improving both inference throughput and memory. Expanding on this, we propose \ours, a novel method that replaces guidance methods with a single refinement of the initial noise. This refined noise enables high-quality image generation without guidance, within the same diffusion pipeline. Our noise-refining model leverages efficient noise-space learning, achieving rapid convergence and strong performance with just 50K text-image pairs. We validate its effectiveness across diverse metrics and analyze how refined noise can eliminate the need for guidance. See our project page: https://cvlab-kaist.github.io/NoiseRefine/.
Cite
@article{arxiv.2412.03895,
title = {A Noise is Worth Diffusion Guidance},
author = {Donghoon Ahn and Jiwon Kang and Sanghyun Lee and Jaewon Min and Minjae Kim and Wooseok Jang and Hyoungwon Cho and Sayak Paul and SeonHwa Kim and Eunju Cha and Kyong Hwan Jin and Seungryong Kim},
journal= {arXiv preprint arXiv:2412.03895},
year = {2024}
}
Comments
Project page: https://cvlab-kaist.github.io/NoiseRefine/