CUGE：一个中文语言理解与生成评测基准
计算与语言
2022-06-15 v2
摘要
实现通用语言智能一直是自然语言处理的长久目标，其中标准评测基准起着基础性和指导性作用。我们认为，对于通用语言智能评测，基准本身需要是全面且系统的。为此，我们提出CUGE，一个具有以下特征的中文语言理解与生成评测基准：（1）分层基准框架，其中数据集按语言能力-任务-数据集的层次结构被原则性地筛选与组织。（2）多级评分策略，基于该分层框架提供不同层次的模型性能。为便于CUGE，我们提供了一个可定制的公开排行榜以支持灵活的模型评判标准。在代表性预训练语言模型上的评测结果表明，朝向通用语言智能仍有充足改进空间。CUGE已在cuge.baai.ac.cn公开可用。
引用
@article{arxiv.2112.13610,
title = {CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark},
author = {Yuan Yao and Qingxiu Dong and Jian Guan and Boxi Cao and Zhengyan Zhang and Chaojun Xiao and Xiaozhi Wang and Fanchao Qi and Junwei Bao and Jinran Nie and Zheni Zeng and Yuxian Gu and Kun Zhou and Xuancheng Huang and Wenhao Li and Shuhuai Ren and Jinliang Lu and Chengqiang Xu and Huadong Wang and Guoyang Zeng and Zile Zhou and Jiajun Zhang and Juanzi Li and Minlie Huang and Rui Yan and Xiaodong He and Xiaojun Wan and Xin Zhao and Xu Sun and Yang Liu and Zhiyuan Liu and Xianpei Han and Erhong Yang and Zhifang Sui and Maosong Sun},
journal= {arXiv preprint arXiv:2112.13610},
year = {2022}
}
备注
We add two new datasets, including grammatical error correction dataset YACLC from Beijing Language and Culture University, and reading comprehension dataset GCRC from Shanxi University, and also improve the description consistency of all datasets