We propose SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates. The model can be trained with various degrees of supervision: 1) self-supervised by the re-projection photometric error (completely unsupervised), 2) supervised by ego-motion (camera motion), or 3) supervised by depth (e.g., as provided by RGBD sensors). SfM-Net extracts meaningful depth estimates and successfully estimates frame-to-frame camera rotations and translations. It often successfully segments the moving objects in the scene, even though such supervision is never provided.
@article{arxiv.1704.07804,
title = {SfM-Net: Learning of Structure and Motion from Video},
author = {Sudheendra Vijayanarasimhan and Susanna Ricco and Cordelia Schmid and Rahul Sukthankar and Katerina Fragkiadaki},
journal= {arXiv preprint arXiv:1704.07804},
year = {2017}
}