DAT: Dialogue-Aware Transformer with Modality-Group Fusion for Human Engagement Estimation
Abstract
Engagement estimation plays a crucial role in understanding human social behaviors, attracting increasing research interests in fields such as affective computing and human-computer interaction. In this paper, we propose a Dialogue-Aware Transformer framework (DAT) with Modality-Group Fusion (MGF), which relies solely on audio-visual input and is language-independent, for estimating human engagement in conversations. Specifically, our method employs a modality-group fusion strategy that independently fuses audio and visual features within each modality for each person before inferring the entire audio-visual content. This strategy significantly enhances the model's performance and robustness. Additionally, to better estimate the target participant's engagement levels, the introduced Dialogue-Aware Transformer considers both the participant's behavior and cues from their conversational partners. Our method was rigorously tested in the Multi-Domain Engagement Estimation Challenge held by MultiMediate'24, demonstrating notable improvements in engagement-level regression precision over the baseline model. Notably, our approach achieves a CCC score of 0.76 on the NoXi Base test set and an average CCC of 0.64 across the NoXi Base, NoXi-Add, and MPIIGI test sets.
Keywords
Cite
@article{arxiv.2410.08470,
title = {DAT: Dialogue-Aware Transformer with Modality-Group Fusion for Human Engagement Estimation},
author = {Jia Li and Yangchen Yu and Yin Chen and Yu Zhang and Peng Jia and Yunbo Xu and Ziqiang Li and Meng Wang and Richang Hong},
journal= {arXiv preprint arXiv:2410.08470},
year = {2024}
}
Comments
1st Place on the NoXi Base dataset in the Multi-Domain Engagement Estimation Challenge held by MultiMediate 24, accepted by ACM Multimedia 2024. The source code is available at \url{https://github.com/MSA-LMC/DAT}