每一步都在演化：规模化强化学习用于万亿参数思考模型
计算与语言
2025-10-28 v2 人工智能
摘要
我们提出了 Ring-1T，这是首个开源的、具有万亿参数规模的思考模型。它拥有 1 万亿个总参数，每个 token 激活约 500 亿个参数。训练如此规模的模型引入了前所未有的挑战，包括训练-推理不一致、rollout 处理效率低下以及 RL 系统瓶颈。为此，我们首次提出三项相互关联的创新措施：(1) IcePop 通过 token 级别的差异遮蔽和裁剪来稳定 RL 训练，解决训练-推理不匹配导致的不稳定问题；(2) C3PO++ 在 token 预算约束下动态划分 rollout，提高资源利用率，实现高时间效率；(3) ASystem 是一个高性能 RL 框架，旨�突破阻碍万亿参数模型训练的系统性瓶颈。Ring-1T 在关键基准测试上取得突破性成绩：AIME-2025 得分 93.4、HMMT-2025 得分 86.72、CodeForces 得分 2088、ARC-AGI-1 得分 55.94。值得注意的是，它在 IMO-2025 上取得银 medal 级别的成绩，凸显其卓越的推理能力。通过向社区开放完整的 1T 参数 MoE 模型，我们为研究社区提供了直接获取前沿推理能力的渠道。这一贡献标志着在大规模推理智能民主化方面迈出了重要步伐，为开源模型性能树立了新的基准。
引用
@article{arxiv.2510.18855,
title = {Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model},
author = {Ling Team and Anqi Shen and Baihui Li and Bin Hu and Bin Jing and Cai Chen and Chao Huang and Chao Zhang and Chaokun Yang and Cheng Lin and Chengyao Wen and Congqi Li and Deng Zhao and Dingbo Yuan and Donghai You and Fagui Mao and Fanzhuang Meng and Feng Xu and Guojie Li and Guowei Wang and Hao Dai and Haonan Zheng and Hong Liu and Jia Guo and Jiaming Liu and Jian Liu and Jianhao Fu and Jiannan Shi and Jianwen Wang and Jianxin Lai and Jin Yang and Jun Mei and Jun Zhou and Junbo Zhao and Junping Zhao and Kuan Xu and Le Su and Lei Chen and Li Tang and Liang Jiang and Liangcheng Fu and Lianhao Xu and Linfeng Shi and Lisha Liao and Longfei Zheng and Meng Li and Mingchun Chen and Qi Zuo and Qiang Cheng and Qianggang Cao and Qitao Shi and Quanrui Guo and Senlin Zhu and Shaofei Wang and Shaomian Zheng and Shuaicheng Li and Shuwei Gu and Siba Chen and Tao Wu and Tao Zhang and Tianyu Zhang and Tianyu Zhou and Tiwei Bie and Tongkai Yang and Wang Hong and Wang Ren and Weihua Chen and Wenbo Yu and Wengang Zheng and Xiangchun Wang and Xiaodong Yan and Xiaopei Wan and Xin Zhao and Xinyu Kong and Xinyu Tang and Xudong Han and Xudong Wang and Xuemin Yang and Xueyu Hu and Yalin Zhang and Yan Sun and Yicheng Shan and Yilong Wang and Yingying Xu and Yongkang Liu and Yongzhen Guo and Yuanyuan Wang and Yuchen Yan and Yuefan Wang and Yuhong Guo and Zehuan Li and Zhankai Xu and Zhe Li and Zhenduo Zhang and Zhengke Gui and Zhenxuan Pan and Zhenyu Huang and Zhenzhong Lan and Zhiqiang Ding and Zhiqiang Zhang and Zhixun Li and Zhizhen Liu and Zihao Wang and Zujie Wen},
journal= {arXiv preprint arXiv:2510.18855},
year = {2025}
}
备注
Technical Report