Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
Computation and Language
2019-11-01 v3 Sound
Audio and Speech Processing
Abstract
Speech recognition technologies are gaining enormous popularity in various industrial applications. However, building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, an unsupervised pre-training method called Masked Predictive Coding is proposed, which can be applied for unsupervised pre-training with Transformer based model. Experiments on HKUST show that using the same training data, we can achieve CER 23.3%, exceeding the best end-to-end model by over 0.2% absolute CER. With more pre-training data, we can further reduce the CER to 21.0%, or a 11.8% relative CER reduction over baseline.
Cite
@article{arxiv.1910.09932,
title = {Improving Transformer-based Speech Recognition Using Unsupervised Pre-training},
author = {Dongwei Jiang and Xiaoning Lei and Wubo Li and Ne Luo and Yuxuan Hu and Wei Zou and Xiangang Li},
journal= {arXiv preprint arXiv:1910.09932},
year = {2019}
}
Comments
Submitted to ICASSP 2020