Recently, automatic video captioning has attracted increasing attention, where the core challenge lies in capturing the key semantic items, like objects and actions as well as their spatial-temporal correlations from the redundant frames and semantic content. To this end, existing works select either the key video clips in a global level~(across multi frames), or key regions within each frame, which, however, neglect the hierarchical order, i.e., key frames first and key regions latter. In this paper, we propose a novel joint-hierarchical attention model for video captioning, which embeds the key clips, the key frames and the key regions jointly into the captioning model in a hierarchical manner. Such a joint-hierarchical attention model first conducts a global selection to identify key frames, followed by a Gumbel sampling operation to identify further key regions based on the key frames, achieving an accurate global-to-local feature representation to guide the captioning. Extensive quantitative evaluations on two public benchmark datasets MSVD and MSR-VTT demonstrates the superiority of the proposed method over the state-of-the-art methods.
@article{arxiv.2203.06663,
title = {Global2Local: A Joint-Hierarchical Attention for Video Captioning},
author = {Chengpeng Dai and Fuhai Chen and Xiaoshuai Sun and Rongrong Ji and Qixiang Ye and Yongjian Wu},
journal= {arXiv preprint arXiv:2203.06663},
year = {2025}
}
Comments
The experiments and the comparisons are out of date. We will advance the framework and the algorithms with large changes