Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion
Abstract
This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or ChatGPT 3.5, due to a predominant focus on mainstream languages (e.g., English and Chinese). One practical objective for an Arabic LLM is to utilize an Arabic-specific vocabulary for the tokenizer that could speed up decoding. However, using a different vocabulary often leads to a degradation of learned knowledge since many words are initially out-of-vocabulary (OOV) when training starts. Inspired by the vocabulary learning during Second Language (Arabic) Acquisition for humans, the released AraLLaMA employs progressive vocabulary expansion, which is implemented by a modified BPE algorithm that progressively extends the Arabic subwords in its dynamic vocabulary during training, thereby balancing the OOV ratio at every stage. The ablation study demonstrated the effectiveness of Progressive Vocabulary Expansion. Moreover, AraLLaMA achieves decent performance comparable to the best Arabic LLMs across a variety of Arabic benchmarks. Models, training data, benchmarks, and codes will be all open-sourced.
Keywords
Cite
@article{arxiv.2412.12310,
title = {Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion},
author = {Jianqing Zhu and Huang Huang and Zhihang Lin and Juhao Liang and Zhengyang Tang and Khalid Almubarak and Abdulmohsen Alharthik and Bang An and Juncai He and Xiangbo Wu and Fei Yu and Junying Chen and Zhuoheng Ma and Yuhao Du and He Zhang and Emad A. Alghamdi and Lian Zhang and Ruoyu Sun and Haizhou Li and Benyou Wang and Jinchao Xu},
journal= {arXiv preprint arXiv:2412.12310},
year = {2025}
}