Understanding dynamic hand motions and actions from egocentric RGB videos is a fundamental yet challenging task due to self-occlusion and ambiguity. To address occlusion and ambiguity, we develop a transformer-based framework to exploit temporal information for robust estimation. Noticing the different temporal granularity of and the semantic correlation between hand pose estimation and action recognition, we build a network hierarchy with two cascaded transformer encoders, where the first one exploits the short-term temporal cue for hand pose estimation, and the latter aggregates per-frame pose and object information over a longer time span to recognize the action. Our approach achieves competitive results on two first-person hand action benchmarks, namely FPHA and H2O. Extensive ablation studies verify our design choices.
@article{arxiv.2209.09484,
title = {Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos},
author = {Yilin Wen and Hao Pan and Lei Yang and Jia Pan and Taku Komura and Wenping Wang},
journal= {arXiv preprint arXiv:2209.09484},
year = {2023}
}
Comments
Accepted by CVPR 2023; Project page: https://fylwen.github.io/htt.html