In this paper, we propose a bilevel joint unsupervised and supervised training (BL-JUST) framework for automatic speech recognition. Compared to the conventional pre-training and fine-tuning strategy which is a disconnected two-stage process, BL-JUST tries to optimize an acoustic model such that it simultaneously minimizes both the unsupervised and supervised loss functions. Because BL-JUST seeks matched local optima of both loss functions, acoustic representations learned by the acoustic model strike a good balance between being generic and task-specific. We solve the BL-JUST problem using penalty-based bilevel gradient descent and evaluate the trained deep neural network acoustic models on various datasets with a variety of architectures and loss functions. We show that BL-JUST can outperform the widely-used pre-training and fine-tuning strategy and some other popular semi-supervised techniques.
@article{arxiv.2412.08548,
title = {Bilevel Joint Unsupervised and Supervised Training for Automatic Speech Recognition},
author = {Xiaodong Cui and A F M Saif and Songtao Lu and Lisha Chen and Tianyi Chen and Brian Kingsbury and George Saon},
journal= {arXiv preprint arXiv:2412.08548},
year = {2024}
}
Comments
Accepted by IEEE/ACM Transactions on Audio, Speech and Language Processing