The Multimodal Emotion Recognition challenge MER2024 focuses on recognizing emotions using audio, language, and visual signals. In this paper, we present our submission solutions for the Semi-Supervised Learning Sub-Challenge (MER2024-SEMI), which tackles the issue of limited annotated data in emotion recognition. Firstly, to address the class imbalance, we adopt an oversampling strategy. Secondly, we propose a modality representation combinatorial contrastive learning (MR-CCL) framework on the trimodal input data to establish robust initial models. Thirdly, we explore a self-training approach to expand the training set. Finally, we enhance prediction robustness through a multi-classifier weighted soft voting strategy. Our proposed method is validated to be effective on the MER2024-SEMI Challenge, achieving a weighted average F-score of 88.25% and ranking 6th on the leaderboard. Our project is available at https://github.com/WooyoohL/MER2024-SEMI.
@article{arxiv.2409.04447,
title = {Leveraging Contrastive Learning and Self-Training for Multimodal Emotion Recognition with Limited Labeled Samples},
author = {Qi Fan and Yutong Li and Yi Xin and Xinyu Cheng and Guanglai Gao and Miao Ma},
journal= {arXiv preprint arXiv:2409.04447},
year = {2024}
}