Can Language Models Learn to Listen?
Abstract
We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words. Given an input transcription of the speaker's words with their timestamps, our approach autoregressively predicts a response of a listener: a sequence of listener facial gestures, quantized using a VQ-VAE. Since gesture is a language component, we propose treating the quantized atomic motion elements as additional language token inputs to a transformer-based large language model. Initializing our transformer with the weights of a language model pre-trained only on text results in significantly higher quality listener responses than training a transformer from scratch. We show that our generated listener motion is fluent and reflective of language semantics through quantitative metrics and a qualitative user study. In our evaluation, we analyze the model's ability to utilize temporal and semantic aspects of spoken text. Project page: https://people.eecs.berkeley.edu/~evonne_ng/projects/text2listen/
Cite
@article{arxiv.2308.10897,
title = {Can Language Models Learn to Listen?},
author = {Evonne Ng and Sanjay Subramanian and Dan Klein and Angjoo Kanazawa and Trevor Darrell and Shiry Ginosar},
journal= {arXiv preprint arXiv:2308.10897},
year = {2023}
}
Comments
ICCV 2023; Project page: https://people.eecs.berkeley.edu/~evonne_ng/projects/text2listen/