Joint Persian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT
Computation and Language
2020-10-29 v2
Abstract
Words are properly segmented in the Persian writing system; in practice, however, these writing rules are often neglected, resulting in single words being written disjointedly and multiple words written without any white spaces between them. This paper addresses the problems of word segmentation and zero-width non-joiner (ZWNJ) recognition in Persian, which we approach jointly as a sequence labeling problem. We achieved a macro-averaged F1-score of 92.40% on a carefully collected corpus of 500 sentences with a high level of difficulty.
Cite
@article{arxiv.2010.00287,
title = {Joint Persian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT},
author = {Ehsan Doostmohammadi and Minoo Nassajian and Adel Rahimi},
journal= {arXiv preprint arXiv:2010.00287},
year = {2020}
}