SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model

Yi-Jen Shih; Hsuan-Fu Wang; Heng-Jui Chang; Layne Berry; Hung-yi Lee; David Harwath

SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model

Computation and Language 2022-10-26 v2 Sound Audio and Speech Processing

Authors: Yi-Jen Shih , Hsuan-Fu Wang , Heng-Jui Chang , Layne Berry , Hung-yi Lee , David Harwath

Abstract

Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.

Keywords

vision-language models contrastive language-image pre-training image captioning

Cite

@article{arxiv.2210.00705,
  title  = {SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model},
  author = {Yi-Jen Shih and Hsuan-Fu Wang and Heng-Jui Chang and Layne Berry and Hung-yi Lee and David Harwath},
  journal= {arXiv preprint arXiv:2210.00705},
  year   = {2022}
}

Comments

Accepted to IEEE SLT 2022

SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model

Abstract

Keywords

Cite

Comments

Related papers