This paper addresses challenges of Natural Language Processing (NLP) on non-canonical multilingual data in which two or more languages are mixed. It refers to code-switching which has become more popular in our daily life and therefore obtains an increasing amount of attention from the research community. We report our experience that cov- ers not only core NLP tasks such as normalisation, language identification, language modelling, part-of-speech tagging and dependency parsing but also more downstream ones such as machine translation and automatic speech recognition. We highlight and discuss the key problems for each of the tasks with supporting examples from different language pairs and relevant previous work.
@article{arxiv.1610.02213,
title = {Challenges of Computational Processing of Code-Switching},
author = {Özlem Çetinoğlu and Sarah Schulz and Ngoc Thang Vu},
journal= {arXiv preprint arXiv:1610.02213},
year = {2016}
}
Comments
Will appear in the Proceedings of the 2nd Workshop on Computational Approaches to Linguistic Code Switching @EMNLP, 2016