SemAlignVC: Enhancing zero-shot timbre conversion using semantic alignment
Abstract
Zero-shot voice conversion (VC) synthesizes speech in a target speaker's voice while preserving linguistic and paralinguistic content. However, timbre leakage-where source speaker traits persist-remains a challenge, especially in neural codec and LLM-based VC, where quantized representations entangle speaker identity with content. We introduce SemAlignVC, an architecture designed to prevent timbre leakage using SemAlign, a novel method that aligns text and audio representations to ensure speaker-independent semantic encoding. This disentangled representation conditions an autoregressive transformer for high-fidelity conversion without explicit speaker embeddings. Experiments show SemAlignVC significantly reduces timbre leakage, outperforming baselines in speaker timbre similarity, intelligibility, and naturalness, making it a robust, privacy-preserving, and generalizable VC solution. Audio samples can be accessed at https://shivammehta25.github.io/SemAlignVC/
Keywords
Cite
@article{arxiv.2507.09070,
title = {SemAlignVC: Enhancing zero-shot timbre conversion using semantic alignment},
author = {Shivam Mehta and Yingru Liu and Zhenyu Tang and Kainan Peng and Vimal Manohar and Shun Zhang and Mike Seltzer and Qing He and Mingbo Ma},
journal= {arXiv preprint arXiv:2507.09070},
year = {2025}
}
Comments
6 pages, 2 figures, Accepted at the ISCA Speech Synthesis Workshop (SSW) 2025