DisCo-Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec
Abstract
Codec-based language models (LMs) have revolutionized text-to-speech (TTS). However, standard codecs entangle timbre and prosody, which hinders independent control in continuation-based LMs. To tackle this challenge, we propose DisCo-Speech, a zero-shot controllable TTS framework featuring a disentangled speech codec (DisCodec) and an LM-based generator. The core component DisCodec employs a two-stage design: 1) tri-factor disentanglement to separate speech into content, prosody, and timbre subspaces via parallel encoders and hybrid losses; and 2) fusion and reconstruction that merges content and prosody into unified content-prosody tokens suitable for LM prediction, while jointly optimizing reconstruction to address the disentanglement-reconstruction trade-off. This allows the LM to perform prosodic continuation from a style prompt while the decoder injects target timbre, enabling flexible zero-shot control. Experiments demonstrate that DisCo-Speech achieves competitive voice cloning and superior zero-shot prosody control. By resolving the core entanglement at the codec level, DisCo-Speech provides a robust foundation for controllable speech synthesis.
Keywords
Cite
@article{arxiv.2512.13251,
title = {DisCo-Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec},
author = {Tao Li and Wenshuo Ge and Zhichao Wang and Zihao Cui and Yong Ma and Yingying Gao and Chao Deng and Shilei Zhang and Junlan Feng},
journal= {arXiv preprint arXiv:2512.13251},
year = {2026}
}
Comments
Updated with 6,000 hours of additional training data and improved performance. Expanded appendix with ablation studies, training objectives, and hyperparameter settings for better reproducibility. Audio and code links included