SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing

Abstract

Image diffusion models have been utilized in various tasks, such as text-to-image generation and controllable im-age synthesis. Recent research has introduced tuning meth-ods that make subtle adjustments to the original models, yielding promising results in specific adaptations of foun-dational generative diffusion models. Rather than modi-fying the main backbone of the diffusion model, we delve into the role of skip connection in U-Net and reveal that hi-erarchical features aggregating long-distance information across encoder and decoder make a significant impact on the content and quality of image generation. Based on the observation, we propose an efficient generative tuning framework, dubbed SCEdit, which integrates and edits Skip Connection using a lightweight tuning module named SC-Tuner. Furthermore, the proposed framework allows for straightforward extension to controllable image syn-thesis by injecting different conditions with Controllable SC-Tuner, simplifying and unifying the network design for multi-condition inputs. Our SCEdit substantially reduces training parameters, memory usage, and computational ex-pense due to its lightweight tuners, with backward propa-gation only passing to the decoder blocks. Extensive exper-iments conducted on text-to-image generation and control-lable image synthesis tasks demonstrate the superiority of our method in terms of efficiency and performance. Project page: https://scedit.github.io/