In This Environment, As That Speaker: A Text-Driven Framework for Multi-Attribute Speech Conversion
Abstract
We propose TES-VC (Text-driven Environment and Speaker controllable Voice Conversion), a text-driven voice conversion framework with independent control of speaker timbre and environmental acoustics. TES-VC processes simultaneous text inputs for target voice and environment, accurately generating speech matching described timbre/environment while preserving source content. Trained on synthetic data with decoupled vocal/environment features via latent diffusion modeling, our method eliminates interference between attributes. The Retrieval-Based Timbre Control (RBTC) module enables precise manipulation using abstract descriptions without paired data. Experiments confirm TES-VC effectively generates contextually appropriate speech in both timbre and environment with high content retention and superior controllability which demonstrates its potential for widespread applications.
Cite
@article{arxiv.2506.07036,
title = {In This Environment, As That Speaker: A Text-Driven Framework for Multi-Attribute Speech Conversion},
author = {Jiawei Jin and Zhihan Yang and Yixuan Zhou and Zhiyong Wu},
journal= {arXiv preprint arXiv:2506.07036},
year = {2025}
}
Comments
Accepted by Interspeech2025