Chinese word segmentation (CWS) is an important task for Chinese NLP. Recently, many neural network based methods have been proposed for CWS. However, these methods require a large number of labeled sentences for model training, and usually cannot utilize the useful information in Chinese dictionary. In this paper, we propose two methods to exploit the dictionary information for CWS. The first one is based on pseudo labeled data generation, and the second one is based on multi-task learning. The experimental results on two benchmark datasets validate that our approach can effectively improve the performance of Chinese word segmentation, especially when training data is insufficient.
@article{arxiv.1807.05849,
title = {Neural Chinese Word Segmentation with Dictionary Knowledge},
author = {Junxin Liu and Fangzhao Wu and Chuhan Wu and Yongfeng Huang and Xing Xie},
journal= {arXiv preprint arXiv:1807.05849},
year = {2018}
}
Comments
This paper has been accepted by The Seventh CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC 2018)