The pre-training of masked language models (MLMs) consumes massive computation to achieve good results on downstream NLP tasks, resulting in a large carbon footprint. In the vanilla MLM, the virtual tokens, [MASK]s, act as placeholders and gather the contextualized information from unmasked tokens to restore the corrupted information. It raises the question of whether we can append [MASK]s at a later layer, to reduce the sequence length for earlier layers and make the pre-training more efficient. We show: (1) [MASK]s can indeed be appended at a later layer, being disentangled from the word embedding; (2) The gathering of contextualized information from unmasked tokens can be conducted with a few layers. By further increasing the masking rate from 15% to 50%, we can pre-train RoBERTa-base and RoBERTa-large from scratch with only 78% and 68% of the original computational budget without any degradation on the GLUE benchmark. When pre-training with the original budget, our method outperforms RoBERTa for 6 out of 8 GLUE tasks, on average by 0.4%.
@article{arxiv.2211.04898,
title = {Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token},
author = {Baohao Liao and David Thulke and Sanjika Hewavitharana and Hermann Ney and Christof Monz},
journal= {arXiv preprint arXiv:2211.04898},
year = {2022}
}
Comments
Code available at: https://github.com/BaohaoLiao/3ml