Collecting high-quality studio recordings of audio is challenging, which limits the language coverage of text-to-speech (TTS) systems. This paper proposes a framework for scaling a multilingual TTS model to 100+ languages using found data without supervision. The proposed framework combines speech-text encoder pretraining with unsupervised training using untranscribed speech and unspoken text data sources, thereby leveraging massively multilingual joint speech and text representation learning. Without any transcribed speech in a new language, this TTS model can generate intelligible speech in >30 unseen languages (CER difference of <10% to ground truth). With just 15 minutes of transcribed, found data, we can reduce the intelligibility difference to 1% or less from the ground-truth, and achieve naturalness scores that match the ground-truth in several languages.
@article{arxiv.2402.18932,
title = {Extending Multilingual Speech Synthesis to 100+ Languages without Transcribed Data},
author = {Takaaki Saeki and Gary Wang and Nobuyuki Morioka and Isaac Elias and Kyle Kastner and Fadi Biadsy and Andrew Rosenberg and Bhuvana Ramabhadran and Heiga Zen and Françoise Beaufays and Hadar Shemtov},
journal= {arXiv preprint arXiv:2402.18932},
year = {2024}
}
Comments
To appear in ICASSP 2024. Demo page: https://google.github.io/tacotron/publications/extending_tts/