We propose a novel task-agnostic in-domain pre-training method that sits between generic pre-training and fine-tuning. Our approach selectively masks in-domain keywords, i.e., words that provide a compact representation of the target domain. We identify such keywords using KeyBERT (Grootendorst, 2020). We evaluate our approach using six different settings: three datasets combined with two distinct pre-trained language models (PLMs). Our results reveal that the fine-tuned PLMs adapted using our in-domain pre-training strategy outperform PLMs that used in-domain pre-training with random masking as well as those that followed the common pre-train-then-fine-tune paradigm. Further, the overhead of identifying in-domain keywords is reasonable, e.g., 7-15% of the pre-training time (for two epochs) for BERT Large (Devlin et al., 2019).
@article{arxiv.2307.07160,
title = {Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords},
author = {Shahriar Golchin and Mihai Surdeanu and Nazgol Tavabi and Ata Kiapour},
journal= {arXiv preprint arXiv:2307.07160},
year = {2023}
}
Comments
final version: accepted at ACL'23 RepL4NLP. arXiv admin note: text overlap with arXiv:2208.12367