Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour. However, most existing methods for hand-object reconstruction from RGB either assume pre-scanned object templates or heavily rely on limited 3D hand-object data, restricting their ability to scale and generalize to more unconstrained interaction settings. To this end, we introduce HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can reconstruct disentangled 3D hand and object from 2D images. We also further incorporate hand-object constraints to improve hand-object poses and consequently the reconstruction quality. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings. Moreover, we qualitatively show its robustness in reconstructing from in-the-wild videos. Code: https://github.com/zc-alexfan/hold
@article{arxiv.2311.18448,
title = {HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video},
author = {Zicong Fan and Maria Parelli and Maria Eleni Kadoglou and Muhammed Kocabas and Xu Chen and Michael J. Black and Otmar Hilliges},
journal= {arXiv preprint arXiv:2311.18448},
year = {2023}
}