Reinforcement learning (RL) has emerged as a critical paradigm for post-training Vision-Language-Action (VLA) models, enabling embodied agents to adapt and improve through environmental interaction. However, existing RL frameworks for VLAs inherit synchronous design principles from traditional LLM training, treating entire rollouts as indivisible units and alternating strictly between data collection and policy optimization. This fundamentally mismatches the unique characteristics of VLA training, as physical simulators introduce highly variable, resource-intensive latencies. To address this, we introduce RL-VLA3, a fully asynchronous distributed RL framework that enables fine-grained asynchronous interaction between simulation, inference, and training components through dynamic batching schedulers and flexible environment sharding strategies. Extensive experiments across diverse simulation backends, VLA architectures, and RL algorithms demonstrate that RL-VLA3 achieves throughput improvements of up to 85.2\% over synchronous baselines while maintaining identical sample efficiency, with scalability validated from 8 to 256 GPUs. To our knowledge, RL-VLA3 is the first fully asynchronous RL training framework tailored specifically for the system-level challenges of VLA training.
@article{arxiv.2602.05765,
title = {RL-VLA$^3$: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training},
author = {Haoran Sun and Yongjian Guo and Zhong Guan and Shuai Di and Xiaodong Bai and Jing Long and Tianyun Zhao and Mingxi Luo and Hongke Zhao and Likang Wu and Xiaotie Deng and Xu Chu and Xi Xiao and Sheng Wen and Yicheng Gong and Junwu Xiong},
journal= {arXiv preprint arXiv:2602.05765},
year = {2026}
}