Remarkable progress has been made in 3D reconstruction of rigid structures from a video or a collection of images. However, it is still challenging to reconstruct nonrigid structures from RGB inputs, due to its under-constrained nature. While template-based approaches, such as parametric shape models, have achieved great success in modeling the "closed world" of known object categories, they cannot well handle the "open-world" of novel object categories or outlier shapes. In this work, we introduce a template-free approach to learn 3D shapes from a single video. It adopts an analysis-by-synthesis strategy that forward-renders object silhouette, optical flow, and pixel values to compare with video observations, which generates gradients to adjust the camera, shape and motion parameters. Without using a category-specific shape template, our method faithfully reconstructs nonrigid 3D structures from videos of human, animals, and objects of unknown classes. Code will be available at lasr-google.github.io .
@article{arxiv.2105.02976,
title = {LASR: Learning Articulated Shape Reconstruction from a Monocular Video},
author = {Gengshan Yang and Deqing Sun and Varun Jampani and Daniel Vlasic and Forrester Cole and Huiwen Chang and Deva Ramanan and William T. Freeman and Ce Liu},
journal= {arXiv preprint arXiv:2105.02976},
year = {2021}
}