With read-aloud speech synthesis achieving high naturalness scores, there is a growing research interest in synthesising spontaneous speech. However, human spontaneous face-to-face conversation has both spoken and non-verbal aspects (here, co-speech gestures). Only recently has research begun to explore the benefits of jointly synthesising these two modalities in a single system. The previous state of the art used non-probabilistic methods, which fail to capture the variability of human speech and motion, and risk producing oversmoothing artefacts and sub-optimal synthesis quality. We present the first diffusion-based probabilistic model, called Diff-TTSG, that jointly learns to synthesise speech and gestures together. Our method can be trained on small datasets from scratch. Furthermore, we describe a set of careful uni- and multi-modal subjective tests for evaluating integrated speech and gesture synthesis systems, and use them to validate our proposed approach. Please see https://shivammehta25.github.io/Diff-TTSG/ for video examples, data, and code.
@article{arxiv.2306.09417,
title = {Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis},
author = {Shivam Mehta and Siyang Wang and Simon Alexanderson and Jonas Beskow and Éva Székely and Gustav Eje Henter},
journal= {arXiv preprint arXiv:2306.09417},
year = {2023}
}
Comments
7 pages, 2 figures, presented at the ISCA Speech Synthesis Workshop (SSW) 2023