Multimodal large language models (MLLMs) are designed to process and integrate information from multiple sources, such as text, speech, images, and videos. Despite its success in language understanding, it is critical to evaluate the performance of downstream tasks for better human-centric applications. This paper assesses the application of MLLMs with 5 crucial abilities for affective computing, spanning from visual affective tasks and reasoning tasks. The results show that \gpt has high accuracy in facial action unit recognition and micro-expression detection while its general facial expression recognition performance is not accurate. We also highlight the challenges of achieving fine-grained micro-expression recognition and the potential for further study and demonstrate the versatility and potential of \gpt for handling advanced tasks in emotion recognition and related fields by integrating with task-related agents for more complex tasks, such as heart rate estimation through signal processing. In conclusion, this paper provides valuable insights into the potential applications and challenges of MLLMs in human-centric computing. Our interesting examples are at https://github.com/EnVision-Research/GPT4Affectivity.
@article{arxiv.2403.05916,
title = {GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing},
author = {Hao Lu and Xuesong Niu and Jiyao Wang and Yin Wang and Qingyong Hu and Jiaqi Tang and Yuting Zhang and Kaishen Yuan and Bin Huang and Zitong Yu and Dengbo He and Shuiguang Deng and Hao Chen and Yingcong Chen and Shiguang Shan},
journal= {arXiv preprint arXiv:2403.05916},
year = {2024}
}