How to boost speech pre-training with textual data is an unsolved problem due to the fact that speech and text are very different modalities with distinct characteristics. In this paper, we propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation. Specifically, we introduce two alternative discrete tokenizers to bridge the speech and text modalities, including phoneme-unit and hidden-unit tokenizers, which can be trained using a small amount of paired speech-text data. Based on the trained tokenizers, we convert the unlabeled speech and text data into tokens of phoneme units or hidden units. The pre-training objective is designed to unify the speech and the text into the same discrete semantic space with a unified Transformer network. We evaluate SpeechLM on various spoken language processing tasks including speech recognition, speech translation, and universal representation evaluation framework SUPERB, demonstrating significant improvements on content-related tasks. Code and models are available at https://aka.ms/SpeechLM.
@article{arxiv.2209.15329,
title = {SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data},
author = {Ziqiang Zhang and Sanyuan Chen and Long Zhou and Yu Wu and Shuo Ren and Shujie Liu and Zhuoyuan Yao and Xun Gong and Lirong Dai and Jinyu Li and Furu Wei},
journal= {arXiv preprint arXiv:2209.15329},
year = {2023}
}
Comments
We have corrected the errors in the pre-training data for SpeechLM-P Base models, new results are updated