Inspired by the fact that different modalities in videos carry complementary information, we propose a Multimodal Semantic Attention Network(MSAN), which is a new encoder-decoder framework incorporating multimodal semantic attributes for video captioning. In the encoding phase, we detect and generate multimodal semantic attributes by formulating it as a multi-label classification problem. Moreover, we add auxiliary classification loss to our model that can obtain more effective visual features and high-level multimodal semantic attribute distributions for sufficient video encoding. In the decoding phase, we extend each weight matrix of the conventional LSTM to an ensemble of attribute-dependent weight matrices, and employ attention mechanism to pay attention to different attributes at each time of the captioning process. We evaluate algorithm on two popular public benchmarks: MSVD and MSR-VTT, achieving competitive results with current state-of-the-art across six evaluation metrics.
@article{arxiv.1905.02963,
title = {Multimodal Semantic Attention Network for Video Captioning},
author = {Liang Sun and Bing Li and Chunfeng Yuan and Zhengjun Zha and Weiming Hu},
journal= {arXiv preprint arXiv:1905.02963},
year = {2019}
}
Comments
6 pages, 4 figures, accepted by IEEE International Conference on Multimedia and Expo (ICME) 2019