English

Measuring Massive Multitask Chinese Understanding

Computation and Language 2026-05-28 v3 Artificial Intelligence

Abstract

The development of large-scale Chinese language models is flourishing, yet there is a lack of corresponding capability assessments. Therefore, we propose a test to measure the multitask accuracy of large Chinese language models. This test encompasses four major domains, including medicine, law, psychology, and education, with 15 subtasks in medicine and 8 subtasks in education. We found that the best-performing models in the zero-shot setting outperformed the worst-performing models by nearly 18.6 percentage points on average. Across the four major domains, the highest average zero-shot accuracy of all models is 0.512. In the subdomains, only the GPT-3.5-turbo model achieved a zero-shot accuracy of 0.693 in clinical medicine, which was the highest accuracy among all models across all subtasks. All models performed poorly in the legal domain, with the highest zero-shot accuracy reaching only 0.239. By comprehensively evaluating the breadth and depth of knowledge across multiple disciplines, this test can more accurately identify the shortcomings of the models.

Keywords

Cite

@article{arxiv.2304.12986,
  title  = {Measuring Massive Multitask Chinese Understanding},
  author = {Hui Zeng},
  journal= {arXiv preprint arXiv:2304.12986},
  year   = {2026}
}
R2 v1 2026-06-28T10:17:31.203Z