Ultra-lightweight Neural Differential DSP Vocoder For High Quality Speech Synthesis
Abstract
Neural vocoders model the raw audio waveform and synthesize high-quality audio, but even the highly efficient ones, like MB-MelGAN and LPCNet, fail to run real-time on a low-end device like a smartglass. A pure digital signal processing (DSP) based vocoder can be implemented via lightweight fast Fourier transforms (FFT), and therefore, is a magnitude faster than any neural vocoder. A DSP vocoder often gets a lower audio quality due to consuming over-smoothed acoustic model predictions of approximate representations for the vocal tract. In this paper, we propose an ultra-lightweight differential DSP (DDSP) vocoder that uses a jointly optimized acoustic model with a DSP vocoder, and learns without an extracted spectral feature for the vocal tract. The model achieves audio quality comparable to neural vocoders with a high average MOS of 4.36 while being efficient as a DSP vocoder. Our C++ implementation, without any hardware-specific optimization, is at 15 MFLOPS, surpasses MB-MelGAN by 340 times in terms of FLOPS, and achieves a vocoder-only RTF of 0.003 and overall RTF of 0.044 while running single-threaded on a 2GHz Intel Xeon CPU.
Cite
@article{arxiv.2401.10460,
title = {Ultra-lightweight Neural Differential DSP Vocoder For High Quality Speech Synthesis},
author = {Prabhav Agrawal and Thilo Koehler and Zhiping Xiu and Prashant Serai and Qing He},
journal= {arXiv preprint arXiv:2401.10460},
year = {2024}
}
Comments
Accepted for ICASSP 2024