Cross-modal learning of video and text plays a key role in Video Question Answering (VideoQA). In this paper, we propose a visual-text attention mechanism to utilize the Contrastive Language-Image Pre-training (CLIP) trained on lots of general domain language-image pairs to guide the cross-modal learning for VideoQA. Specifically, we first extract video features using a TimeSformer and text features using a BERT from the target application domain, and utilize CLIP to extract a pair of visual-text features from the general-knowledge domain through the domain-specific learning. We then propose a Cross-domain Learning to extract the attention information between visual and linguistic features across the target domain and general domain. The set of CLIP-guided visual-text features are integrated to predict the answer. The proposed method is evaluated on MSVD-QA and MSRVTT-QA datasets, and outperforms state-of-the-art methods.
@article{arxiv.2303.03131,
title = {Video Question Answering Using CLIP-Guided Visual-Text Attention},
author = {Shuhong Ye and Weikai Kong and Chenglin Yao and Jianfeng Ren and Xudong Jiang},
journal= {arXiv preprint arXiv:2303.03131},
year = {2023}
}
Comments
Submitted to the 2023 IEEE International Conference on Image Processing (ICIP 2023)