Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations
Abstract
Speech restoration (SR) is a task of converting degraded speech signals into high-quality ones. In this study, we propose a robust SR model called Miipher, and apply Miipher to a new SR application: increasing the amount of high-quality training data for speech generation by converting speech samples collected from the Web to studio-quality. To make our SR model robust against various degradation, we use (i) a speech representation extracted from w2v-BERT for the input feature, and (ii) a text representation extracted from transcripts via PnG-BERT as a linguistic conditioning feature. Experiments show that Miipher (i) is robust against various audio degradation and (ii) enable us to train a high-quality text-to-speech (TTS) model from restored speech samples collected from the Web. Audio samples are available at our demo page: google.github.io/df-conformer/miipher/
Cite
@article{arxiv.2303.01664,
title = {Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations},
author = {Yuma Koizumi and Heiga Zen and Shigeki Karita and Yifan Ding and Kohei Yatabe and Nobuyuki Morioka and Yu Zhang and Wei Han and Ankur Bapna and Michiel Bacchiani},
journal= {arXiv preprint arXiv:2303.01664},
year = {2023}
}
Comments
Accepted to WASPAA 2023