PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning
Abstract
Neural speech codecs have achieved strong performance in low-bitrate compression, but residual vector quantization (RVQ) often suffers from unstable training and ineffective decomposition, limiting reconstruction quality and efficiency. We propose PURE Codec (Progressive Unfolding of Residual Entropy), a novel framework that guides multi-stage quantization using a pre-trained speech enhancement model. The first quantization stage reconstructs low-entropy, denoised speech embeddings, while subsequent stages encode residual high-entropy components. This design improves training stability significantly. Experiments demonstrate that PURE consistently outperforms conventional RVQ-based codecs in reconstruction and downstream speech language model-based text-to-speech, particularly under noisy training conditions.
Keywords
Cite
@article{arxiv.2511.22687,
title = {PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning},
author = {Jiatong Shi and Haoran Wang and William Chen and Chenda Li and Wangyou Zhang and Jinchuan Tian and Shinji Watanabe},
journal= {arXiv preprint arXiv:2511.22687},
year = {2025}
}
Comments
Accepted by ASRU2025