We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training strategy has demonstrated sample-efficiency to pretrain models at the scale of hundreds of millions of parameters. In this work, we conduct a comprehensive empirical study, and propose a recipe, namely "Model generated dEnoising TRaining Objective" (METRO), which incorporates some of the best modeling techniques developed recently to speed up, stabilize, and enhance pretrained language models without compromising model effectiveness. The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks. More importantly, METRO-LM are efficient in that they often outperform previous large models with significantly smaller model sizes and lower pretraining cost.
@article{arxiv.2204.06644,
title = {METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals},
author = {Payal Bajaj and Chenyan Xiong and Guolin Ke and Xiaodong Liu and Di He and Saurabh Tiwary and Tie-Yan Liu and Paul Bennett and Xia Song and Jianfeng Gao},
journal= {arXiv preprint arXiv:2204.06644},
year = {2022}
}
Comments
Update details in scaled initialization and add acknowledgement