Neural Neural Textures Make Sim2Real Consistent
Abstract
Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and \emph{neural neural textures}. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic-looking learnable neural textures. By settling on a particular texture for the objects in a scene, we ensure consistency between frames statelessly. Unlike previous algorithms, TRITON is not limited to camera movements -- it can handle the movement of objects as well, making it useful for downstream tasks such as robotic manipulation.
Cite
@article{arxiv.2206.13500,
title = {Neural Neural Textures Make Sim2Real Consistent},
author = {Ryan Burgert and Jinghuan Shang and Xiang Li and Michael Ryoo},
journal= {arXiv preprint arXiv:2206.13500},
year = {2022}
}
Comments
9 pages, 10 figures (without references or appendix); 16 pages, 16 figures (with appendix)