Audio-Visual Speech Enhancement for Spatial Audio - Spatial-VisualVoice and the MAVE Database
Abstract
Audio-visual speech enhancement (AVSE) has been found to be particularly useful at low signal-to-noise (SNR) ratios due to the immunity of the visual features to acoustic noise. However, a significant gap exists in AVSE methods tailored to enhance spatial audio under low-SNR conditions. The latter is of growing interest with augmented reality applications. To address this gap, we present a multi-channel AVSE framework based on VisualVoice that leverages spatial cues from microphone arrays and visual information for enhancing the target speaker in noisy environments. We also introduce MAVe, a novel database containing multi-channel audio-visual signals in controlled, reproducible room conditions across a wide range of SNR levels. Experiments demonstrate that the proposed method consistently achieves significant gains in SI-SDR, STOI, and PESQ, particularly in low SNRs. Binaural signal analysis further confirms the preservation of spatial cues and intelligibility.
Cite
@article{arxiv.2510.16437,
title = {Audio-Visual Speech Enhancement for Spatial Audio - Spatial-VisualVoice and the MAVE Database},
author = {Danielle Yaffe and Ferdinand Campe and Prachi Sharma and Dorothea Kolossa and Boaz Rafaely},
journal= {arXiv preprint arXiv:2510.16437},
year = {2025}
}