In recent years, the size of pre-trained language models (PLMs) has grown by leaps and bounds. However, efficiency issues of these large-scale PLMs limit their utilization in real-world scenarios. We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference. (1) We introduce knowledge inheritance to accelerate the pre-training process by exploiting existing PLMs instead of training models from scratch. (2) We explore the best practice of prompt tuning with large-scale PLMs. Compared with conventional fine-tuning, prompt tuning significantly reduces the number of task-specific parameters. (3) We implement a new inference toolkit, namely InfMoE, for using large-scale PLMs with limited computational resources. Based on our cost-effective pipeline, we pre-train two models: an encoder-decoder bilingual model with 11 billion parameters (CPM-2) and its corresponding MoE version with 198 billion parameters. In our experiments, we compare CPM-2 with mT5 on downstream tasks. Experimental results show that CPM-2 has excellent general language intelligence. Moreover, we validate the efficiency of InfMoE when conducting inference of large-scale models having tens of billions of parameters on a single GPU. All source code and model parameters are available at https://github.com/TsinghuaAI/CPM.
@article{arxiv.2106.10715,
title = {CPM-2: Large-scale Cost-effective Pre-trained Language Models},
author = {Zhengyan Zhang and Yuxian Gu and Xu Han and Shengqi Chen and Chaojun Xiao and Zhenbo Sun and Yuan Yao and Fanchao Qi and Jian Guan and Pei Ke and Yanzheng Cai and Guoyang Zeng and Zhixing Tan and Zhiyuan Liu and Minlie Huang and Wentao Han and Yang Liu and Xiaoyan Zhu and Maosong Sun},
journal= {arXiv preprint arXiv:2106.10715},
year = {2021}
}