Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems designed to accelerate RL post-training still suffer from low resource utilization and limited scalability. We present ROLL Flash, a system that extends ROLL with native support for asynchronous RL post-training. ROLL Flash is built upon two core design principles: fine-grained parallelism and rollout-train decoupling. Guided by these principles, ROLL Flash provides flexible programming interfaces that enable a fully asynchronous training architecture and support efficient rollout mechanisms, including queue scheduling and environment-level asynchronous execution. Through comprehensive theoretical analysis and extensive experiments, we demonstrate that ROLL Flash significantly improves resource utilization and scalability over synchronous RL post-training. ROLL Flash achieves up to 2.24x speedup on RLVR tasks and 2.72x on agentic tasks, using the same GPU budget as synchronous baselines. Furthermore, we implement several popular off-policy algorithms and verify that asynchronous training can achieve performance on par with synchronous training.
@article{arxiv.2510.11345,
title = {Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony},
author = {Han Lu and Zichen Liu and Shaopan Xiong and Yancheng He and Wei Gao and Yanan Wu and Weixun Wang and Jiashun Liu and Yang Li and Haizhou Zhao and Ju Huang and Siran Yang and Xiaoyang Li and Yijia Luo and Zihe Liu and Ling Pan and Junchi Yan and Wei Wang and Wenbo Su and Jiamang Wang and Lin Qu and Bo Zheng},
journal= {arXiv preprint arXiv:2510.11345},
year = {2025}
}