We introduce Llama-3-Motif, a language model consisting of 102 billion parameters, specifically designed to enhance Korean capabilities while retaining strong performance in English. Developed on the Llama 3 architecture, Llama-3-Motif employs advanced training techniques, including LlamaPro and Masked Structure Growth, to effectively scale the model without altering its core Transformer architecture. Using the MoAI platform for efficient training across hyperscale GPU clusters, we optimized Llama-3-Motif using a carefully curated dataset that maintains a balanced ratio of Korean and English data. Llama-3-Motif shows decent performance on Korean-specific benchmarks, outperforming existing models and achieving results comparable to GPT-4.
@article{arxiv.2509.03972,
title = {Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study},
author = {Junghwan Lim and Gangwon Jo and Sungmin Lee and Jiyoung Park and Dongseok Kim and Jihwan Kim and Junhyeok Lee and Wai Ting Cheung and Dahye Choi and Kibong Choi and Jaeyeon Huh and Beomgyu Kim and Jangwoong Kim and Taehyun Kim and Haesol Lee and Jeesoo Lee and Dongpin Oh and Changseok Song and Daewon Suh},
journal= {arXiv preprint arXiv:2509.03972},
year = {2025}
}