Unlike mainstream languages (such as English and French), low-resource languages often suffer from a lack of expert-annotated corpora and benchmark resources that make it hard to apply state-of-the-art techniques directly. In this paper, we alleviate this scarcity problem for the low-resourced Filipino language in two ways. First, we introduce a new benchmark language modeling dataset in Filipino which we call WikiText-TL-39. Second, we show that language model finetuning techniques such as BERT and ULMFiT can be used to consistently train robust classifiers in low-resource settings, experiencing at most a 0.0782 increase in validation error when the number of training examples is decreased from 10K to 1K while finetuning using a privately-held sentiment dataset.
@article{arxiv.1907.00409,
title = {Evaluating Language Model Finetuning Techniques for Low-resource Languages},
author = {Jan Christian Blaise Cruz and Charibeth Cheng},
journal= {arXiv preprint arXiv:1907.00409},
year = {2019}
}
Comments
Pretrained models and datasets available at https://github.com/jcblaisecruz02/Tagalog-BERT