Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition
Abstract
The sparsely-gated Mixture of Experts (MoE) can magnify a network capacity with a little computational complexity. In this work, we investigate how multi-lingual Automatic Speech Recognition (ASR) networks can be scaled up with a simple routing algorithm in order to achieve better accuracy. More specifically, we apply the sparsely-gated MoE technique to two types of networks: Sequence-to-Sequence Transformer (S2S-T) and Transformer Transducer (T-T). We demonstrate through a set of ASR experiments on multiple language data that the MoE networks can reduce the relative word error rates by 16.3% and 4.6% with the S2S-T and T-T, respectively. Moreover, we thoroughly investigate the effect of the MoE on the T-T architecture in various conditions: streaming mode, non-streaming mode, the use of language ID and the label decoder with the MoE.
Keywords
Cite
@article{arxiv.2112.05820,
title = {Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition},
author = {Kenichi Kumatani and Robert Gmyr and Felipe Cruz Salinas and Linquan Liu and Wei Zuo and Devang Patel and Eric Sun and Yu Shi},
journal= {arXiv preprint arXiv:2112.05820},
year = {2022}
}