Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability.
@article{arxiv.2010.10938,
title = {What makes multilingual BERT multilingual?},
author = {Chi-Liang Liu and Tsung-Yuan Hsu and Yung-Sung Chuang and Hung-yi Lee},
journal= {arXiv preprint arXiv:2010.10938},
year = {2020}
}
Comments
arXiv admin note: substantial text overlap with arXiv:2004.09205