Universal Speech Content Factorization
Abstract
We propose Universal Speech Content Factorization (USCF), a simple and invertible linear method for extracting a low-rank speech representation in which speaker timbre is suppressed while phonetic content is preserved. USCF extends Speech Content Factorization, a closed-set voice conversion (VC) method, to an open-set setting by learning a universal speech-to-content mapping via least-squares optimization and deriving speaker-specific transformations from only a few seconds of target speech. We show through embedding analysis that USCF effectively removes speaker-dependent variation. As a zero-shot VC system, USCF achieves competitive intelligibility, naturalness, and speaker similarity compared to methods that require substantially more target-speaker data or additional neural training. Finally, we demonstrate that as a training-efficient timbre-disentangled speech feature, USCF features can serve as the acoustic representation for training timbre-prompted text-to-speech models. Speech samples and code are publicly available.
Cite
@article{arxiv.2603.08977,
title = {Universal Speech Content Factorization},
author = {Henry Li Xinyuan and Zexin Cai and Lin Zhang and Leibny Paola García-Perera and Berrak Sisman and Sanjeev Khudanpur and Nicholas Andrews and Matthew Wiesner},
journal= {arXiv preprint arXiv:2603.08977},
year = {2026}
}