Monocular 4D human-object interaction (HOI) reconstruction - recovering a moving human and a manipulated object from a single RGB video - remains challenging due to depth ambiguity and frequent occlusions. Existing methods often rely on multi-stage pipelines or iterative optimization, leading to high inference latency, failing to meet real-time requirements, and susceptibility to error accumulation. To address these limitations, we propose THO, an end-to-end Spatial-Temporal Transformer that predicts human motion and coordinated object motion in a forward fashion from the given video and 3D template. THO achieves this by leveraging spatial-temporal HOI tuple priors. Spatial priors exploit contact-region proximity to infer occluded object features from human cues, while temporal priors capture cross-frame kinematic correlations to refine object representations and enforce physical coherence. Extensive experiments demonstrate that THO operates at an inference speed of 31.5 FPS on a single RTX 4090 GPU, achieving a >600x speedup over prior optimization-based methods while simultaneously improving reconstruction accuracy and temporal consistency. The project page is available at: https://nianheng.github.io/THO-project/
@article{arxiv.2603.14435,
title = {End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction},
author = {Haoyu Zhang and Wei Zhai and Yuhang Yang and Yang Cao and Zheng-Jun Zha},
journal= {arXiv preprint arXiv:2603.14435},
year = {2026}
}
Comments
23 pages, 7 figures. The project page is available at: https://nianheng.github.io/THO-project/