End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
Abstract
We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard LibriSpeech dataset, and leverage additional unlabeled data from LibriVox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.
Cite
@article{arxiv.1911.08460,
title = {End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures},
author = {Gabriel Synnaeve and Qiantong Xu and Jacob Kahn and Tatiana Likhomanenko and Edouard Grave and Vineel Pratap and Anuroop Sriram and Vitaliy Liptchinsky and Ronan Collobert},
journal= {arXiv preprint arXiv:1911.08460},
year = {2020}
}
Comments
Published at the workshop on Self-supervision in Audio and Speech (SAS) at the 37th International Conference on Machine Learning (ICML 2020), Vienna, Austria