We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time.
@article{arxiv.2501.01715,
title = {Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision},
author = {Alberta Longhini and Marcel Büsching and Bardienus P. Duisterhof and Jens Lundell and Jeffrey Ichnowski and Mårten Björkman and Danica Kragic},
journal= {arXiv preprint arXiv:2501.01715},
year = {2025}
}
Comments
Accepted at the 8th Conference on Robot Learning (CoRL 2024). Code and videos available at: kth-rpl.github.io/cloth-splatting