DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
Abstract
The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.
Cite
@article{arxiv.2401.02954,
title = {DeepSeek LLM: Scaling Open-Source Language Models with Longtermism},
author = {DeepSeek-AI and : and Xiao Bi and Deli Chen and Guanting Chen and Shanhuang Chen and Damai Dai and Chengqi Deng and Honghui Ding and Kai Dong and Qiushi Du and Zhe Fu and Huazuo Gao and Kaige Gao and Wenjun Gao and Ruiqi Ge and Kang Guan and Daya Guo and Jianzhong Guo and Guangbo Hao and Zhewen Hao and Ying He and Wenjie Hu and Panpan Huang and Erhang Li and Guowei Li and Jiashi Li and Yao Li and Y. K. Li and Wenfeng Liang and Fangyun Lin and A. X. Liu and Bo Liu and Wen Liu and Xiaodong Liu and Xin Liu and Yiyuan Liu and Haoyu Lu and Shanghao Lu and Fuli Luo and Shirong Ma and Xiaotao Nie and Tian Pei and Yishi Piao and Junjie Qiu and Hui Qu and Tongzheng Ren and Zehui Ren and Chong Ruan and Zhangli Sha and Zhihong Shao and Junxiao Song and Xuecheng Su and Jingxiang Sun and Yaofeng Sun and Minghui Tang and Bingxuan Wang and Peiyi Wang and Shiyu Wang and Yaohui Wang and Yongji Wang and Tong Wu and Y. Wu and Xin Xie and Zhenda Xie and Ziwei Xie and Yiliang Xiong and Hanwei Xu and R. X. Xu and Yanhong Xu and Dejian Yang and Yuxiang You and Shuiping Yu and Xingkai Yu and B. Zhang and Haowei Zhang and Lecong Zhang and Liyue Zhang and Mingchuan Zhang and Minghua Zhang and Wentao Zhang and Yichao Zhang and Chenggang Zhao and Yao Zhao and Shangyan Zhou and Shunfeng Zhou and Qihao Zhu and Yuheng Zou},
journal= {arXiv preprint arXiv:2401.02954},
year = {2024}
}