We propose a novel approach to Vietnamese word segmentation. Our approach is based on the Single Classification Ripple Down Rules methodology (Compton and Jansen, 1990), where rules are stored in an exception structure and new rules are only added to correct segmentation errors given by existing rules. Experimental results on the benchmark Vietnamese treebank show that our approach outperforms previous state-of-the-art approaches JVnSegmenter, vnTokenizer, DongDu and UETsegmenter in terms of both accuracy and performance speed. Our code is open-source and available at: https://github.com/datquocnguyen/RDRsegmenter.
Cite
@article{arxiv.1709.06307,
title = {A Fast and Accurate Vietnamese Word Segmenter},
author = {Dat Quoc Nguyen and Dai Quoc Nguyen and Thanh Vu and Mark Dras and Mark Johnson},
journal= {arXiv preprint arXiv:1709.06307},
year = {2017}
}
Comments
In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018), to appear