This paper introduces ChineseVideoBench, a pioneering benchmark specifically designed for evaluating Multimodal Large Language Models (MLLMs) in Chinese Video Question Answering. The growing demand for sophisticated video analysis capabilities highlights the critical need for comprehensive, culturally-aware evaluation frameworks. ChineseVideoBench addresses this gap by providing a robust dataset and tailored evaluation metrics, enabling rigorous assessment of state-of-the-art MLLMs on complex Chinese video content. Specifically, ChineseVideoBench comprises 8 main classes and 12 sub-classes, encompassing tasks that demand both deep video understanding and nuanced Chinese linguistic and cultural awareness. Our empirical evaluations reveal that ChineseVideoBench presents a significant challenge to current MLLMs. Among the models assessed, Gemini 2.5 Pro achieves the highest performance with an overall score of 77.9%, while InternVL-38B emerges as the most competitive open-source model.
@article{arxiv.2511.18399,
title = {ChineseVideoBench: Benchmarking Multi-modal Large Models for Chinese Video Question Answering},
author = {Yuxiang Nie and Han Wang and Yongjie Ye and Haiyang Yu and Weitao Jia and Tao Zeng and Hao Feng and Xiang Fei and Yang Li and Xiaohui Lv and Guozhi Tang and Jingqun Tang and Jinghui Lu and Zehui Dai and Jiacong Wang and Dingkang Yang and An-Lan Wang and Can Huang},
journal= {arXiv preprint arXiv:2511.18399},
year = {2025}
}