Emotion Recognition in Conversations (ERC) is essential for building empathetic human-machine systems. Existing studies on ERC primarily focus on summarizing the context information in a conversation, however, ignoring the differentiated emotional behaviors within and across different modalities. Designing appropriate strategies that fit the differentiated multi-modal emotional behaviors can produce more accurate emotional predictions. Thus, we propose the DialogueTransformer to explore the differentiated emotional behaviors from the intra- and inter-modal perspectives. For intra-modal, we construct a novel Hierarchical Transformer that can easily switch between sequential and feed-forward structures according to the differentiated context preference within each modality. For inter-modal, we constitute a novel Multi-Grained Interactive Fusion that applies both neuron- and vector-grained feature interactions to learn the differentiated contributions across all modalities. Experimental results show that DialogueTRM outperforms the state-of-the-art by a significant margin on three benchmark datasets.
@article{arxiv.2010.07637,
title = {DialogueTRM: Exploring the Intra- and Inter-Modal Emotional Behaviors in the Conversation},
author = {Yuzhao Mao and Qi Sun and Guang Liu and Xiaojie Wang and Weiguo Gao and Xuan Li and Jianping Shen},
journal= {arXiv preprint arXiv:2010.07637},
year = {2020}
}