Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion
Abstract
Robust audio-visual speech recognition (AVSR) in noisy environments remains challenging, as existing systems struggle to estimate audio reliability and dynamically adjust modality reliance. We propose router-gated cross-modal feature fusion, a novel AVSR framework that adaptively reweights audio and visual features based on token-level acoustic corruption scores. Using an audio-visual feature fusion-based router, our method down-weights unreliable audio tokens and reinforces visual cues through gated cross-attention in each decoder layer. This enables the model to pivot toward the visual modality when audio quality deteriorates. Experiments on LRS3 demonstrate that our approach achieves an 16.51-42.67% relative reduction in word error rate compared to AV-HuBERT. Ablation studies confirm that both the router and gating mechanism contribute to improved robustness under real-world acoustic noise.
Keywords
Cite
@article{arxiv.2508.18734,
title = {Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion},
author = {DongHoon Lim and YoungChae Kim and Dong-Hyun Kim and Da-Hee Yang and Joon-Hyuk Chang},
journal= {arXiv preprint arXiv:2508.18734},
year = {2025}
}
Comments
Accepted to IEEE ASRU 2025