We present our transducer model on Librispeech. We study variants to include an external language model (LM) with shallow fusion and subtract an estimated internal LM. This is justified by a Bayesian interpretation where the transducer model prior is given by the estimated internal LM. The subtraction of the internal LM gives us over 14% relative improvement over normal shallow fusion. Our transducer has a separate probability distribution for the non-blank labels which allows for easier combination with the external LM, and easier estimation of the internal LM. We additionally take care of including the end-of-sentence (EOS) probability of the external LM in the last blank probability which further improves the performance. All our code and setups are published.
@article{arxiv.2104.03006,
title = {Librispeech Transducer Model with Internal Language Model Prior Correction},
author = {Albert Zeyer and André Merboldt and Wilfried Michel and Ralf Schlüter and Hermann Ney},
journal= {arXiv preprint arXiv:2104.03006},
year = {2021}
}