AttaCut: A Fast and Accurate Neural Thai Word Segmenter
Abstract
Word segmentation is a fundamental pre-processing step for Thai Natural Language Processing. The current off-the-shelf solutions are not benchmarked consistently, so it is difficult to compare their trade-offs. We conducted a speed and accuracy comparison of the popular systems on three different domains and found that the state-of-the-art deep learning system is slow and moreover does not use sub-word structures to guide the model. Here, we propose a fast and accurate neural Thai Word Segmenter that uses dilated CNN filters to capture the environment of each character and uses syllable embeddings as features. Our system runs at least 5.6x faster and outperforms the previous state-of-the-art system on some domains. In addition, we develop the first ML-based Thai orthographical syllable segmenter, which yields syllable embeddings to be used as features by the word segmenter.
Cite
@article{arxiv.1911.07056,
title = {AttaCut: A Fast and Accurate Neural Thai Word Segmenter},
author = {Pattarawat Chormai and Ponrawee Prasertsom and Attapol Rutherford},
journal= {arXiv preprint arXiv:1911.07056},
year = {2019}
}
Comments
14 pages, 7 figures, accepted as oral presentation at New in ML Workshop, NeurIPS 2019