English

MPN: Multimodal Parallel Network for Audio-Visual Event Localization

Computer Vision and Pattern Recognition 2021-04-08 v1 Artificial Intelligence Multimedia

Abstract

Audio-visual event localization aims to localize an event that is both audible and visible in the wild, which is a widespread audio-visual scene analysis task for unconstrained videos. To address this task, we propose a Multimodal Parallel Network (MPN), which can perceive global semantics and unmixed local information parallelly. Specifically, our MPN framework consists of a classification subnetwork to predict event categories and a localization subnetwork to predict event boundaries. The classification subnetwork is constructed by the Multimodal Co-attention Module (MCM) and obtains global contexts. The localization subnetwork consists of Multimodal Bottleneck Attention Module (MBAM), which is designed to extract fine-grained segment-level contents. Extensive experiments demonstrate that our framework achieves the state-of-the-art performance both in fully supervised and weakly supervised settings on the Audio-Visual Event (AVE) dataset.

Keywords

Cite

@article{arxiv.2104.02971,
  title  = {MPN: Multimodal Parallel Network for Audio-Visual Event Localization},
  author = {Jiashuo Yu and Ying Cheng and Rui Feng},
  journal= {arXiv preprint arXiv:2104.02971},
  year   = {2021}
}

Comments

IEEE International Conference on Multimedia and Expo (ICME) 2021 Oral

R2 v1 2026-06-24T00:54:53.554Z