Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy
Abstract
Audio-visual target speech extraction (AV-TSE) is one of the enabling technologies in robotics and many audio-visual applications. One of the challenges of AV-TSE is how to effectively utilize audio-visual synchronization information in the process. AV-HuBERT can be a useful pre-trained model for lip-reading, which has not been adopted by AV-TSE. In this paper, we would like to explore the way to integrate a pre-trained AV-HuBERT into our AV-TSE system. We have good reasons to expect an improved performance. To benefit from the inter and intra-modality correlations, we also propose a novel Mask-And-Recover (MAR) strategy for self-supervised learning. The experimental results on the VoxCeleb2 dataset show that our proposed model outperforms the baselines both in terms of subjective and objective metrics, suggesting that the pre-trained AV-HuBERT model provides more informative visual cues for target speech extraction. Furthermore, through a comparative study, we confirm that the proposed Mask-And-Recover strategy is significantly effective.
Keywords
Cite
@article{arxiv.2403.16078,
title = {Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy},
author = {Wenxuan Wu and Xueyuan Chen and Xixin Wu and Haizhou Li and Helen Meng},
journal= {arXiv preprint arXiv:2403.16078},
year = {2024}
}
Comments
Accepted by IJCNN 2024