We propose a self-supervised framework to learn scene representations from video that are automatically delineated into objects and background. Our method relies on moving objects being equivariant with respect to their transformation across frames and the background being constant. After training, we can manipulate and render the scenes in real time to create unseen combinations of objects, transformations, and backgrounds. We show results on moving MNIST with backgrounds.
@article{arxiv.2011.05787,
title = {Learned Equivariant Rendering without Transformation Supervision},
author = {Cinjon Resnick and Or Litany and Hugo Larochelle and Joan Bruna and Kyunghyun Cho},
journal= {arXiv preprint arXiv:2011.05787},
year = {2020}
}
Comments
Workshop on Differentiable Vision, Graphics, and Physics in Machine Learning at NeurIPS 2020