Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction
Abstract
Target speech extraction aims to extract, based on a given conditioning cue, a target speech signal that is corrupted by interfering sources, such as noise or competing speakers. Building upon the achievements of the state-of-the-art (SOTA) time-frequency speaker separation model TF-GridNet, we propose AV-GridNet, a visual-grounded variant that incorporates the face recording of a target speaker as a conditioning factor during the extraction process. Recognizing the inherent dissimilarities between speech and noise signals as interfering sources, we also propose SAV-GridNet, a scenario-aware model that identifies the type of interfering scenario first and then applies a dedicated expert model trained specifically for that scenario. Our proposed model achieves SOTA results on the second COG-MHEAR Audio-Visual Speech Enhancement Challenge, outperforming other models by a significant margin, objectively and in a listening test. We also perform an extensive analysis of the results under the two scenarios.
Cite
@article{arxiv.2310.19644,
title = {Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction},
author = {Zexu Pan and Gordon Wichern and Yoshiki Masuyama and Francois G. Germain and Sameer Khurana and Chiori Hori and Jonathan Le Roux},
journal= {arXiv preprint arXiv:2310.19644},
year = {2023}
}
Comments
Accepted by ASRU 2023