强化学习在大规模推理模型中的综述
计算与语言
2025-10-10 v3 人工智能
机器学习
摘要
本文综述了近期在大型语言模型（LLM）推理任务中应用强化学习（RL）的最新进展。RL在推进LLM能力的前沿方面取得了显著成功，尤其是在解决数学和编码等复杂逻辑任务方面。因此，RL已成为将LLM转化为LRMs（Large Reasoning Models）的基础方法之一。随着该领域的快速发展，进一步扩展RL以服务于人工超智能（ASI）面临的挑战不仅包括计算资源，还包括算法设计、训练数据和基础设施。为此，有必要重审该领域的发展，重新评估其轨迹，并探索提升RL可扩展性的策略。本文特别考察了自DeepSeek-R1发布以来，应用RL于LLMs和LRMs以提升推理能力的研究，包括基础组件、核心问题、训练资源和下游应用，以识别该快速演进领域的未来机遇和方向。我们希望本综述能促进RL在更广泛的推理模型上的未来研究。Github: https://github.com/TsinghuaC3I/Awesome-RL-for-LRMs
引用
@article{arxiv.2509.08827,
title = {A Survey of Reinforcement Learning for Large Reasoning Models},
author = {Kaiyan Zhang and Yuxin Zuo and Bingxiang He and Youbang Sun and Runze Liu and Che Jiang and Yuchen Fan and Kai Tian and Guoli Jia and Pengfei Li and Yu Fu and Xingtai Lv and Yuchen Zhang and Sihang Zeng and Shang Qu and Haozhan Li and Shijie Wang and Yuru Wang and Xinwei Long and Fangfu Liu and Xiang Xu and Jiaze Ma and Xuekai Zhu and Ermo Hua and Yihao Liu and Zonglin Li and Huayu Chen and Xiaoye Qu and Yafu Li and Weize Chen and Zhenzhao Yuan and Junqi Gao and Dong Li and Zhiyuan Ma and Ganqu Cui and Zhiyuan Liu and Biqing Qi and Ning Ding and Bowen Zhou},
journal= {arXiv preprint arXiv:2509.08827},
year = {2025}
}
备注
Fixed typos; added missing and recent citations (117 -> 120 pages)