Aligning diffusion models to downstream tasks often requires finetuning new models or gradient-based guidance at inference time to enable sampling from the reward-tilted posterior. In this work, we explore a simple inference-time gradient-free guidance approach, called controlled denoising (CoDe), that circumvents the need for differentiable guidance functions and model finetuning. CoDe is a blockwise sampling method applied during intermediate denoising steps, allowing for alignment with downstream rewards. Our experiments demonstrate that, despite its simplicity, CoDe offers a favorable trade-off between reward alignment, prompt instruction following, and inference cost, achieving a competitive performance against the state-of-the-art baselines. Our code is available at: https://github.com/anujinho/code.
@article{arxiv.2502.00968,
title = {CoDe: Blockwise Control for Denoising Diffusion Models},
author = {Anuj Singh and Sayak Mukherjee and Ahmad Beirami and Hadi Jamali-Rad},
journal= {arXiv preprint arXiv:2502.00968},
year = {2025}
}