Adaptable Multi-Domain Language Model for Transformer ASR
Abstract
We propose an adapter based multi-domain Transformer based language model (LM) for Transformer ASR. The model consists of a big size common LM and small size adapters. The model can perform multi-domain adaptation with only the small size adapters and its related layers. The proposed model can reuse the full fine-tuned LM which is fine-tuned using all layers of an original model. The proposed LM can be expanded to new domains by adding about 2% of parameters for a first domain and 13% parameters for after second domain. The proposed model is also effective in reducing the model maintenance cost because it is possible to omit the costly and time-consuming common LM pre-training process. Using proposed adapter based approach, we observed that a general LM with adapter can outperform a dedicated music domain LM in terms of word error rate (WER).
Cite
@article{arxiv.2008.06208,
title = {Adaptable Multi-Domain Language Model for Transformer ASR},
author = {Taewoo Lee and Min-Joong Lee and Tae Gyoon Kang and Seokyeoung Jung and Minseok Kwon and Yeona Hong and Jungin Lee and Kyoung-Gu Woo and Ho-Gyeong Kim and Jiseung Jeong and Jihyun Lee and Hosik Lee and Young Sang Choi},
journal= {arXiv preprint arXiv:2008.06208},
year = {2021}
}
Comments
This paper is accepted for presentation at IEEE International Conference on Acoustics, Speech and Signal Processing (IEEE ICASSP), 2021