Recent text-to-image models have achieved impressive results. However, since they require large-scale datasets of text-image pairs, it is impractical to train them on new domains where data is scarce or not labeled. In this work, we propose using large-scale retrieval methods, in particular, efficient k-Nearest-Neighbors (kNN), which offers novel capabilities: (1) training a substantially small and efficient text-to-image diffusion model without any text, (2) generating out-of-distribution images by simply swapping the retrieval database at inference time, and (3) performing text-driven local semantic manipulations while preserving object identity. To demonstrate the robustness of our method, we apply our kNN approach on two state-of-the-art diffusion backbones, and show results on several different datasets. As evaluated by human studies and automatic metrics, our method achieves state-of-the-art results compared to existing approaches that train text-to-image generation models using images only (without paired text data)
@article{arxiv.2204.02849,
title = {KNN-Diffusion: Image Generation via Large-Scale Retrieval},
author = {Shelly Sheynin and Oron Ashual and Adam Polyak and Uriel Singer and Oran Gafni and Eliya Nachmani and Yaniv Taigman},
journal= {arXiv preprint arXiv:2204.02849},
year = {2022}
}