语言模型词汇扩展与初始化方法的实证比较
摘要
语言模型在英语自然语言处理任务中表现出色，但在大多数其他语言方面表现下降。这一问题通常通过不断预训练和微调这些模型来解决所需语言。一个显著问题是原始模型分词器的词汇覆盖有限，导致对新语言缺乏充分表示，迫使需要扩展分词器。与新词汇对应的嵌入初始化 presents a further challenge. Current strategies require cross-lingual embeddings and lack a solid theoretical foundation as well as comparisons with strong baselines. In this paper, we first establish theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V), which does not require cross-lingual embeddings. Our study evaluates different initialization methods for expanding RoBERTa and LLaMA 2 across four languages and five tasks. The results show that CW2V performs equally well or even better than more advanced techniques. Additionally, simpler approaches like multivariate initialization perform on par with these advanced methods indicating that efficient large-scale multilingual continued pretraining can be achieved even with simpler initialization methods. We release our code publicly (https://github.com/AI4Bharat/VocabAdaptation_LLM/tree/CW2V).
引用
@article{arxiv.2407.05841,
title = {An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models},
author = {Nandini Mundra and Aditya Nanda Kishore and Raj Dabre and Ratish Puduppully and Anoop Kunchukuttan and Mitesh M. Khapra},
journal= {arXiv preprint arXiv:2407.05841},
year = {2024}
}
备注
CONLL 2024 (EMNLP 2024)