We investigate how to generate multimodal image outputs, such as RGB, depth, and surface normals, with a single generative model. The challenge is to produce outputs that are realistic, and also consistent with each other. Our solution builds on the StyleGAN3 architecture, with a shared backbone and modality-specific branches in the last layers of the synthesis network, and we propose per-modality fidelity discriminators and a cross-modality consistency discriminator. In experiments on the Stanford2D3D dataset, we demonstrate realistic and consistent generation of RGB, depth, and normal images. We also show a training recipe to easily extend our pretrained model on a new domain, even with a few pairwise data. We further evaluate the use of synthetically generated RGB and depth pairs for training or fine-tuning depth estimators. Code will be available at https://github.com/jessemelpolio/MultimodalGAN.
@article{arxiv.2307.01425,
title = {Consistent Multimodal Generation via A Unified GAN Framework},
author = {Zhen Zhu and Yijun Li and Weijie Lyu and Krishna Kumar Singh and Zhixin Shu and Soeren Pirk and Derek Hoiem},
journal= {arXiv preprint arXiv:2307.01425},
year = {2023}
}