Second pass rescoring is a critical component of competitive automatic speech recognition (ASR) systems. Large language models have demonstrated their ability in using pre-trained information for better rescoring of ASR hypothesis. Discriminative training, directly optimizing the minimum word-error-rate (MWER) criterion typically improves rescoring. In this study, we propose and explore several discriminative fine-tuning schemes for pre-trained LMs. We propose two architectures based on different pooling strategies of output embeddings and compare with probability based MWER. We conduct detailed comparisons between pre-trained causal and bidirectional LMs in discriminative settings. Experiments on LibriSpeech demonstrate that all MWER training schemes are beneficial, giving additional gains upto 8.5\% WER. Proposed pooling variants achieve lower latency while retaining most improvements. Finally, our study concludes that bidirectionality is better utilized with discriminative training.
@article{arxiv.2310.06248,
title = {Discriminative Speech Recognition Rescoring with Pre-trained Language Models},
author = {Prashanth Gurunath Shivakumar and Jari Kolehmainen and Yile Gu and Ankur Gandhe and Ariya Rastrow and Ivan Bulyko},
journal= {arXiv preprint arXiv:2310.06248},
year = {2023}
}