Pre-training with Meta Learning for Chinese Word Segmentation
Abstract
Recent researches show that pre-trained models (PTMs) are beneficial to Chinese Word Segmentation (CWS). However, PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge and ignoring the discrepancy between pre-training tasks and downstream CWS tasks. In this paper, we propose a CWS-specific pre-trained model METASEG, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task. Empirical results show that METASEG could utilize common prior segmentation knowledge from different existing criteria and alleviate the discrepancy between pre-trained models and downstream CWS tasks. Besides, METASEG can achieve new state-of-the-art performance on twelve widely-used CWS datasets and significantly improve model performance in low-resource settings.
Cite
@article{arxiv.2010.12272,
title = {Pre-training with Meta Learning for Chinese Word Segmentation},
author = {Zhen Ke and Liang Shi and Songtao Sun and Erli Meng and Bin Wang and Xipeng Qiu},
journal= {arXiv preprint arXiv:2010.12272},
year = {2021}
}
Comments
Accepted by NAACL 2021