We introduce a way to learn to estimate a scene representation from a single image by predicting a low-dimensional subspace of optical flow for each training example, which encompasses the variety of possible camera and object movement. Supervision is provided by a novel loss which measures the distance between this predicted flow subspace and an observed optical flow. This provides a new approach to learning scene representation tasks, such as monocular depth prediction or instance segmentation, in an unsupervised fashion using in-the-wild input videos without requiring camera poses, intrinsics, or an explicit multi-view stereo step. We evaluate our method in multiple settings, including an indoor depth prediction task where it achieves comparable performance to recent methods trained with more supervision.
@article{arxiv.2112.01502,
title = {Dimensions of Motion: Monocular Prediction through Flow Subspaces},
author = {Richard Strong Bowen and Richard Tucker and Ramin Zabih and Noah Snavely},
journal= {arXiv preprint arXiv:2112.01502},
year = {2022}
}
Comments
Project page at https://dimensions-of-motion.github.io/