Object-oriented maps are important for scene understanding since they jointly capture geometry and semantics, allow individual instantiation and meaningful reasoning about objects. We introduce FroDO, a method for accurate 3D reconstruction of object instances from RGB video that infers object location, pose and shape in a coarse-to-fine manner. Key to FroDO is to embed object shapes in a novel learnt space that allows seamless switching between sparse point cloud and dense DeepSDF decoding. Given an input sequence of localized RGB frames, FroDO first aggregates 2D detections to instantiate a category-aware 3D bounding box per object. A shape code is regressed using an encoder network before optimizing shape and pose further under the learnt shape priors using sparse and dense shape representations. The optimization uses multi-view geometric, photometric and silhouette losses. We evaluate on real-world datasets, including Pix3D, Redwood-OS, and ScanNet, for single-view, multi-view, and multi-object reconstruction.
@article{arxiv.2005.05125,
title = {FroDO: From Detections to 3D Objects},
author = {Kejie Li and Martin Rünz and Meng Tang and Lingni Ma and Chen Kong and Tanner Schmidt and Ian Reid and Lourdes Agapito and Julian Straub and Steven Lovegrove and Richard Newcombe},
journal= {arXiv preprint arXiv:2005.05125},
year = {2020}
}
Comments
To be published in CVPR 2020. The first two authors contributed equally