SparseVSR: Lightweight and Noise Robust Visual Speech Recognition
Abstract
Recent advances in deep neural networks have achieved unprecedented success in visual speech recognition. However, there remains substantial disparity between current methods and their deployment in resource-constrained devices. In this work, we explore different magnitude-based pruning techniques to generate a lightweight model that achieves higher performance than its dense model equivalent, especially under the presence of visual noise. Our sparse models achieve state-of-the-art results at 10% sparsity on the LRS3 dataset and outperform the dense equivalent up to 70% sparsity. We evaluate our 50% sparse model on 7 different visual noise types and achieve an overall absolute improvement of more than 2% WER compared to the dense equivalent. Our results confirm that sparse networks are more resistant to noise than dense networks.
Cite
@article{arxiv.2307.04552,
title = {SparseVSR: Lightweight and Noise Robust Visual Speech Recognition},
author = {Adriana Fernandez-Lopez and Honglie Chen and Pingchuan Ma and Alexandros Haliassos and Stavros Petridis and Maja Pantic},
journal= {arXiv preprint arXiv:2307.04552},
year = {2023}
}
Comments
Accepted to Interspeech 2023