Attention-based sequence-to-sequence models for automatic speech recognition jointly train an acoustic model, language model, and alignment mechanism. Thus, the language model component is only trained on transcribed audio-text pairs. This leads to the use of shallow fusion with an external language model at inference time. Shallow fusion refers to log-linear interpolation with a separately trained language model at each step of the beam search. In this work, we investigate the behavior of shallow fusion across a range of conditions: different types of language models, different decoding units, and different tasks. On Google Voice Search, we demonstrate that the use of shallow fusion with a neural LM with wordpieces yields a 9.1% relative word error rate reduction (WERR) over our competitive attention-based sequence-to-sequence model, obviating the need for second-pass rescoring.
@article{arxiv.1712.01996,
title = {An analysis of incorporating an external language model into a sequence-to-sequence model},
author = {Anjuli Kannan and Yonghui Wu and Patrick Nguyen and Tara N. Sainath and Zhifeng Chen and Rohit Prabhavalkar},
journal= {arXiv preprint arXiv:1712.01996},
year = {2017}
}