NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models

Abstract

In training-free conditional generative tasks, diffusion models utilize differentiable loss functions to steer the generative reverse process, necessitating modifications to sampling algorithms like DDPM and DDIM. However, such adjustments likely reduce flexibility and reliability. In this paper, we propose NoiseCtrl, a sampling-algorithm-agnostic technique for controlled image generation. Essentially, diffusion models generate denoised results zt−1 by adding a predicted mean µt with random noise ϵt. NoiseCtrl specifically adjusts the random noise while leaving the underlying sampling algorithms unchanged. At each step t, NoiseCtrl converts the unconditional Gaussian noise into conditional noise $\varepsilon _t^\prime $ by substituting the isotropic Gaussian distribution with the von Mises–Fisher distribution. This substitution introduces a directional focus while preserving the randomness required for conditional image generation. Thanks to this non-intrusive design, NoiseCtrl is straightforward to integrate and has been extensively validated through experiments, demonstrating its adaptability for different diffusion algorithms and superior performance across various conditional generation tasks.