In this paper, we address the performance degradation of efficient diffusion models by introducing Multi-architecturE Multi-Expert diffusion models (MEME). We identify the need for tailored operations at different time-steps in diffusion processes and leverage this insight to create compact yet high-performing models. MEME assigns distinct architectures to different time-step intervals, balancing convolution and self-attention operations based on observed frequency characteristics. We also introduce a soft interval assignment strategy for comprehensive training. Empirically, MEME operates 3.3 times faster than baselines while improving image generation quality (FID scores) by 0.62 (FFHQ) and 0.37 (CelebA). Though we validate the effectiveness of assigning more optimal architecture per time-step, where efficient models outperform the larger models, we argue that MEME opens a new design choice for diffusion models that can be easily applied in other scenarios, such as large multi-expert models.
@article{arxiv.2306.04990,
title = {Multi-Architecture Multi-Expert Diffusion Models},
author = {Yunsung Lee and Jin-Young Kim and Hyojun Go and Myeongho Jeong and Shinhyeok Oh and Seungtaek Choi},
journal= {arXiv preprint arXiv:2306.04990},
year = {2023}
}
Comments
To be published in the AAAI 2024 Proceedings Main Track