Kimi k1.5：规模化强化学习
人工智能
2025-06-04 v4 机器学习
摘要
基于下一个标记预测的语言模型预训练已证明有效，用于扩大计算规模，但仅限于可用训练数据的量。规模化强化学习( RL)为人工智能的持续改进开辟了新的轴向，具有大型语言模型( LLM)通过学习奖励驱动的探索来扩大训练数据的潜力。然而，过去的公开工作未产生具竞争力的结果。鉴于此，我们报告了Kimi k1.5的训练实践，这是我们最新采用RL训练的多模态LLM，包括其RL训练技术、多模态数据配方以及基础设施优化。长上下文扩展和改进的策略优化方法是我们方法的关键要素，该方法构建了一个简洁有效的RL框架，而无需依赖蒙特卡洛树搜索、价值函数和过程奖励模型等更复杂的技术。值得注意的是，我们的系统在多个基准和模态上实现了领先的推理性能——例如在AIME上达到77.5分，在MATH 500上达到96.2分，在Codeforces上达到94百分位，在MathVista上达到74.9分，匹配OpenAI的o1。此外，我们present了有效的long2short方法，该方法利用long-CoT技术改进short-CoT模型，产生了领先的short-CoT推理结果——例如在AIME上达到60.8分，在MATH500上达到94.6分，在LiveCodeBench上达到47.3分，超过现有的short-CoT模型如GPT-4o和Claude Sonnet 3.5 by a large margin(最多高出550%)。
引用
@article{arxiv.2501.12599,
title = {Kimi k1.5: Scaling Reinforcement Learning with LLMs},
author = {Kimi Team and Angang Du and Bofei Gao and Bowei Xing and Changjiu Jiang and Cheng Chen and Cheng Li and Chenjun Xiao and Chenzhuang Du and Chonghua Liao and Chuning Tang and Congcong Wang and Dehao Zhang and Enming Yuan and Enzhe Lu and Fengxiang Tang and Flood Sung and Guangda Wei and Guokun Lai and Haiqing Guo and Han Zhu and Hao Ding and Hao Hu and Hao Yang and Hao Zhang and Haotian Yao and Haotian Zhao and Haoyu Lu and Haoze Li and Haozhen Yu and Hongcheng Gao and Huabin Zheng and Huan Yuan and Jia Chen and Jianhang Guo and Jianlin Su and Jianzhou Wang and Jie Zhao and Jin Zhang and Jingyuan Liu and Junjie Yan and Junyan Wu and Lidong Shi and Ling Ye and Longhui Yu and Mengnan Dong and Neo Zhang and Ningchen Ma and Qiwei Pan and Qucheng Gong and Shaowei Liu and Shengling Ma and Shupeng Wei and Sihan Cao and Siying Huang and Tao Jiang and Weihao Gao and Weimin Xiong and Weiran He and Weixiao Huang and Weixin Xu and Wenhao Wu and Wenyang He and Xianghui Wei and Xianqing Jia and Xingzhe Wu and Xinran Xu and Xinxing Zu and Xinyu Zhou and Xuehai Pan and Y. Charles and Yang Li and Yangyang Hu and Yangyang Liu and Yanru Chen and Yejie Wang and Yibo Liu and Yidao Qin and Yifeng Liu and Ying Yang and Yiping Bao and Yulun Du and Yuxin Wu and Yuzhi Wang and Zaida Zhou and Zhaoji Wang and Zhaowei Li and Zhen Zhu and Zheng Zhang and Zhexu Wang and Zhilin Yang and Zhiqi Huang and Zihao Huang and Ziyao Xu and Zonghan Yang and Zongyu Lin},
journal= {arXiv preprint arXiv:2501.12599},
year = {2025}
}
备注
25 pages