One-shot imitation is the vision of robot programming from a single demonstration, rather than by tedious construction of computer code. We present a practical method for realizing one-shot imitation for manipulation tasks, exploiting modern learning-based optical flow to perform real-time visual servoing. Our approach, which we call FlowControl, continuously tracks a demonstration video, using a specified foreground mask to attend to an object of interest. Using RGB-D observations, FlowControl requires no 3D object models, and is easy to set up. FlowControl inherits great robustness to visual appearance from decades of work in optical flow. We exhibit FlowControl on a range of problems, including ones requiring very precise motions, and ones requiring the ability to generalize.
@article{arxiv.2007.00291,
title = {FlowControl: Optical Flow Based Visual Servoing},
author = {Max Argus and Lukas Hermann and Jon Long and Thomas Brox},
journal= {arXiv preprint arXiv:2007.00291},
year = {2020}
}