Recent work has shown the possibility of training generative models of 3D content from 2D image collections on small datasets corresponding to a single object class, such as human faces, animal faces, or cars. However, these models struggle on larger, more complex datasets. To model diverse and unconstrained image collections such as ImageNet, we present VQ3D, which introduces a NeRF-based decoder into a two-stage vector-quantized autoencoder. Our Stage 1 allows for the reconstruction of an input image and the ability to change the camera position around the image, and our Stage 2 allows for the generation of new 3D scenes. VQ3D is capable of generating and reconstructing 3D-aware images from the 1000-class ImageNet dataset of 1.2 million training images. We achieve an ImageNet generation FID score of 16.8, compared to 69.8 for the next best baseline method.
@article{arxiv.2302.06833,
title = {VQ3D: Learning a 3D-Aware Generative Model on ImageNet},
author = {Kyle Sargent and Jing Yu Koh and Han Zhang and Huiwen Chang and Charles Herrmann and Pratul Srinivasan and Jiajun Wu and Deqing Sun},
journal= {arXiv preprint arXiv:2302.06833},
year = {2023}
}
Comments
15 pages. For visual results, please visit the project webpage at http://kylesargent.github.io/vq3d