CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion
Abstract
Zero-shot voice conversion (VC) aims to convert the original speaker's timbre to any target speaker while keeping the linguistic content. Current mainstream zero-shot voice conversion approaches depend on pre-trained recognition models to disentangle linguistic content and speaker representation. This results in a timbre residue within the decoupled linguistic content and inadequacies in speaker representation modeling. In this study, we propose CoDiff-VC, an end-to-end framework for zero-shot voice conversion that integrates a speech codec and a diffusion model to produce high-fidelity waveforms. Our approach involves employing a single-codebook codec to separate linguistic content from the source speech. To enhance content disentanglement, we introduce Mix-Style layer normalization (MSLN) to perturb the original timbre. Additionally, we incorporate a multi-scale speaker timbre modeling approach to ensure timbre consistency and improve voice detail similarity. To improve speech quality and speaker similarity, we introduce dual classifier-free guidance, providing both content and timbre guidance during the generation process. Objective and subjective experiments affirm that CoDiff-VC significantly improves speaker similarity, generating natural and higher-quality speech.
Cite
@article{arxiv.2411.18918,
title = {CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion},
author = {Yuke Li and Xinfa Zhu and Hanzhao Li and JiXun Yao and WenJie Tian and XiPeng Yang and YunLin Chen and Zhifei Li and Lei Xie},
journal= {arXiv preprint arXiv:2411.18918},
year = {2024}
}