VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing
Abstract
Deep learning has brought significant improvements to the field of cross-modal representation learning. For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired, emphasizing the semantic content of the text modality while de-emphasizing the paralinguistic information of the speech modality. We propose a method called "Vector Quantized Contrastive Token-Acoustic Pre-training (VQ-CTAP)", which uses the cross-modal aligned sequence transcoder to bring text and speech into a joint multimodal space, learning how to connect text and speech at the frame level. The proposed VQ-CTAP is a paradigm for cross-modal sequence representation learning, offering a promising solution for fine-grained generation and recognition tasks in speech processing. The VQ-CTAP can be directly applied to VC and ASR tasks without fine-tuning or additional structures. We propose a sequence-aware semantic connector, which connects multiple frozen pre-trained modules for the TTS task, exhibiting a plug-and-play capability. We design a stepping optimization strategy to ensure effective model convergence by gradually injecting and adjusting the influence of various loss components. Furthermore, we propose a semantic-transfer-wise paralinguistic consistency loss to enhance representational capabilities, allowing the model to better generalize to unseen data and capture the nuances of paralinguistic information. In addition, VQ-CTAP achieves high-compression speech coding at a rate of 25Hz from 24kHz input waveforms, which is a 960-fold reduction in the sampling rate. The audio demo is available at https://qiangchunyu.github.io/VQCTAP/
Cite
@article{arxiv.2408.05758,
title = {VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing},
author = {Chunyu Qiang and Wang Geng and Yi Zhao and Ruibo Fu and Tao Wang and Cheng Gong and Tianrui Wang and Qiuyu Liu and Jiangyan Yi and Zhengqi Wen and Chen Zhang and Hao Che and Longbiao Wang and Jianwu Dang and Jianhua Tao},
journal= {arXiv preprint arXiv:2408.05758},
year = {2025}
}