Joint Audio and Speech Understanding
Abstract
Humans are surrounded by audio signals that include both speech and non-speech sounds. The recognition and understanding of speech and non-speech audio events, along with a profound comprehension of the relationship between them, constitute fundamental cognitive capabilities. For the first time, we build a machine learning model, called LTU-AS, that has a conceptually similar universal audio perception and advanced reasoning ability. Specifically, by integrating Whisper as a perception module and LLaMA as a reasoning module, LTU-AS can simultaneously recognize and jointly understand spoken text, speech paralinguistics, and non-speech audio events - almost everything perceivable from audio signals.
Cite
@article{arxiv.2309.14405,
title = {Joint Audio and Speech Understanding},
author = {Yuan Gong and Alexander H. Liu and Hongyin Luo and Leonid Karlinsky and James Glass},
journal= {arXiv preprint arXiv:2309.14405},
year = {2023}
}
Comments
Accepted at ASRU 2023. Code, dataset, and pretrained models are at https://github.com/yuangongnd/ltu. Interactive demo at https://huggingface.co/spaces/yuangongfdu/ltu-2