Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training
Abstract
Recently, masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. It usually requires a codebook obtained in an unsupervised way, making it less accurate and difficult to interpret. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance and also the pre-training efficiency, either through decoding with a hybrid ASR system to generate phoneme-level alignments (named PBERT), or performing clustering on the supervised speech features extracted from an end-to-end CTC model (named CTC clustering). Both the hybrid and CTC models are trained on the same small amount of labeled speech as used in fine-tuning. Experiments demonstrate significant superiority of our methods to various SSL and self-training baselines, with up to 17.0% relative WER reduction. Our pre-trained models also show good transferability in a non-ASR speech task.
Cite
@article{arxiv.2206.10125,
title = {Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training},
author = {Chengyi Wang and Yiming Wang and Yu Wu and Sanyuan Chen and Jinyu Li and Shujie Liu and Furu Wei},
journal= {arXiv preprint arXiv:2206.10125},
year = {2022}
}
Comments
To appear in Proc. Interspeech 2022