We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to image and scribble from/to image, and a text guidance, our model automatically understands the underlying task and performs the same task on a new query image following the text guidance. To achieve this, we propose a vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input. The diffusion model is trained jointly over six different tasks using these prompts. The resulting Prompt Diffusion model is the first diffusion-based vision-language foundation model capable of in-context learning. It demonstrates high-quality in-context generation on the trained tasks and generalizes effectively to new, unseen vision tasks with their respective prompts. Our model also shows compelling text-guided image editing results. Our framework aims to facilitate research into in-context learning for computer vision. We share our code and pre-trained models at https://github.com/Zhendong-Wang/Prompt-Diffusion.
@article{arxiv.2305.01115,
title = {In-Context Learning Unlocked for Diffusion Models},
author = {Zhendong Wang and Yifan Jiang and Yadong Lu and Yelong Shen and Pengcheng He and Weizhu Chen and Zhangyang Wang and Mingyuan Zhou},
journal= {arXiv preprint arXiv:2305.01115},
year = {2023}
}