Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, including the use of verifiable reward tasks, enhancements to Group Relative Policy Optimization (GRPO), and practical techniques to improve training stability and generalization. We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains. Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks. To facilitate continued research, we release our model publicly.
@article{arxiv.2507.12507,
title = {Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training},
author = {Mingjie Liu and Shizhe Diao and Jian Hu and Ximing Lu and Xin Dong and Hao Zhang and Alexander Bukharin and Shaokun Zhang and Jiaqi Zeng and Makesh Narsimhan Sreedhar and Gerald Shen and David Mosallanezhad and Di Zhang and Jonas Yang and June Yang and Oleksii Kuchaiev and Guilin Liu and Zhiding Yu and Pavlo Molchanov and Yejin Choi and Jan Kautz and Yi Dong},
journal= {arXiv preprint arXiv:2507.12507},
year = {2025}
}