User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization
Computation and Language
2021-04-09 v1
Abstract
Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.
Cite
@article{arxiv.2104.03523,
title = {User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization},
author = {Shohei Higashiyama and Masao Utiyama and Taro Watanabe and Eiichiro Sumita},
journal= {arXiv preprint arXiv:2104.03523},
year = {2021}
}
Comments
NAACL-HLT 2021