YAYI 2: Multilingual Open-Source Large Language Models
Abstract
As the latest advancements in natural language processing, large language models (LLMs) have achieved human-level language understanding and generation abilities in many real-world tasks, and even have been regarded as a potential path to the artificial general intelligence. To better facilitate research on LLMs, many open-source LLMs, such as Llama 2 and Falcon, have recently been proposed and gained comparable performances to proprietary models. However, these models are primarily designed for English scenarios and exhibit poor performances in Chinese contexts. In this technical report, we propose YAYI 2, including both base and chat models, with 30 billion parameters. YAYI 2 is pre-trained from scratch on a multilingual corpus which contains 2.65 trillion tokens filtered by our pre-training data processing pipeline. The base model is aligned with human values through supervised fine-tuning with millions of instructions and reinforcement learning from human feedback. Extensive experiments on multiple benchmarks, such as MMLU and CMMLU, consistently demonstrate that the proposed YAYI 2 outperforms other similar sized open-source models.
Cite
@article{arxiv.2312.14862,
title = {YAYI 2: Multilingual Open-Source Large Language Models},
author = {Yin Luo and Qingchao Kong and Nan Xu and Jia Cao and Bao Hao and Baoyu Qu and Bo Chen and Chao Zhu and Chenyang Zhao and Donglei Zhang and Fan Feng and Feifei Zhao and Hailong Sun and Hanxuan Yang and Haojun Pan and Hongyu Liu and Jianbin Guo and Jiangtao Du and Jingyi Wang and Junfeng Li and Lei Sun and Liduo Liu and Lifeng Dong and Lili Liu and Lin Wang and Liwen Zhang and Minzheng Wang and Pin Wang and Ping Yu and Qingxiao Li and Rui Yan and Rui Zou and Ruiqun Li and Taiwen Huang and Xiaodong Wang and Xiaofei Wu and Xin Peng and Xina Zhang and Xing Fang and Xinglin Xiao and Yanni Hao and Yao Dong and Yigang Wang and Ying Liu and Yongyu Jiang and Yungan Wang and Yuqi Wang and Zhangsheng Wang and Zhaoxin Yu and Zhen Luo and Wenji Mao and Lei Wang and Dajun Zeng},
journal= {arXiv preprint arXiv:2312.14862},
year = {2023}
}