DeepSeek LLM：以长期主义扩展开源语言模型
计算与语言
2024-01-08 v1 人工智能
机器学习
摘要
开源大语言模型（LLM）的快速发展令人瞩目。然而，以往文献中描述的缩放定律给出了不同的结论，这给 LLM 的扩展蒙上了一层阴影。我们深入研究了缩放定律，并提出了我们独特的发现，这些发现有助于在两种常用的开源配置（7B 和 67B）中扩展大规模模型。在缩放定律的指导下，我们推出了 DeepSeek LLM，这是一个致力于以长远视角推进开源语言模型的项目。为了支持预训练阶段，我们开发了一个目前包含 2 万亿个 token 且仍在不断扩展的数据集。我们进一步对 DeepSeek LLM Base 模型进行监督微调（SFT）和直接偏好优化（DPO），从而创建了 DeepSeek Chat 模型。我们的评估结果表明，DeepSeek LLM 67B 在各种基准测试上超越了 LLaMA-2 70B，尤其是在代码、数学和推理领域。此外，开放式评估表明，DeepSeek LLM 67B Chat 表现出优于 GPT-3.5 的性能。
引用
@article{arxiv.2401.02954,
title = {DeepSeek LLM: Scaling Open-Source Language Models with Longtermism},
author = {DeepSeek-AI and : and Xiao Bi and Deli Chen and Guanting Chen and Shanhuang Chen and Damai Dai and Chengqi Deng and Honghui Ding and Kai Dong and Qiushi Du and Zhe Fu and Huazuo Gao and Kaige Gao and Wenjun Gao and Ruiqi Ge and Kang Guan and Daya Guo and Jianzhong Guo and Guangbo Hao and Zhewen Hao and Ying He and Wenjie Hu and Panpan Huang and Erhang Li and Guowei Li and Jiashi Li and Yao Li and Y. K. Li and Wenfeng Liang and Fangyun Lin and A. X. Liu and Bo Liu and Wen Liu and Xiaodong Liu and Xin Liu and Yiyuan Liu and Haoyu Lu and Shanghao Lu and Fuli Luo and Shirong Ma and Xiaotao Nie and Tian Pei and Yishi Piao and Junjie Qiu and Hui Qu and Tongzheng Ren and Zehui Ren and Chong Ruan and Zhangli Sha and Zhihong Shao and Junxiao Song and Xuecheng Su and Jingxiang Sun and Yaofeng Sun and Minghui Tang and Bingxuan Wang and Peiyi Wang and Shiyu Wang and Yaohui Wang and Yongji Wang and Tong Wu and Y. Wu and Xin Xie and Zhenda Xie and Ziwei Xie and Yiliang Xiong and Hanwei Xu and R. X. Xu and Yanhong Xu and Dejian Yang and Yuxiang You and Shuiping Yu and Xingkai Yu and B. Zhang and Haowei Zhang and Lecong Zhang and Liyue Zhang and Mingchuan Zhang and Minghua Zhang and Wentao Zhang and Yichao Zhang and Chenggang Zhao and Yao Zhao and Shangyan Zhou and Shunfeng Zhou and Qihao Zhu and Yuheng Zou},
journal= {arXiv preprint arXiv:2401.02954},
year = {2024}
}