Audio-Visual Wake Word Spotting System For MISP Challenge 2021
Abstract
This paper presents the details of our system designed for the Task 1 of Multimodal Information Based Speech Processing (MISP) Challenge 2021. The purpose of Task 1 is to leverage both audio and video information to improve the environmental robustness of far-field wake word spotting. In the proposed system, firstly, we take advantage of speech enhancement algorithms such as beamforming and weighted prediction error (WPE) to address the multi-microphone conversational audio. Secondly, several data augmentation techniques are applied to simulate a more realistic far-field scenario. For the video information, the provided region of interest (ROI) is used to obtain visual representation. Then the multi-layer CNN is proposed to learn audio and visual representations, and these representations are fed into our two-branch attention-based network which can be employed for fusion, such as transformer and conformed. The focal loss is used to fine-tune the model and improve the performance significantly. Finally, multiple trained models are integrated by casting vote to achieve our final 0.091 score.
Cite
@article{arxiv.2204.08686,
title = {Audio-Visual Wake Word Spotting System For MISP Challenge 2021},
author = {Yanguang Xu and Jianwei Sun and Yang Han and Shuaijiang Zhao and Chaoyang Mei and Tingwei Guo and Shuran Zhou and Chuandong Xie and Wei Zou and Xiangang Li and Shuran Zhou and Chuandong Xie and Wei Zou and Xiangang Li},
journal= {arXiv preprint arXiv:2204.08686},
year = {2022}
}
Comments
Accepted to ICASSP 2022