Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information. In the context of two real-world robotic tasks, we show:(i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.
@article{arxiv.2210.03701,
title = {VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects},
author = {Youngsun Wi and Andy Zeng and Pete Florence and Nima Fazeli},
journal= {arXiv preprint arXiv:2210.03701},
year = {2022}
}