Improving data utilization efficiency is critical for scaling reinforcement learning (RL) for long-horizon tasks where generating trajectories is expensive. However, the dominant RL methods for LLMs are largely on-policy: they update each batch of data only once, discard it, and then collect fresh samples, resulting in poor sample efficiency. In this work, we explore an alternative value-based RL framework for LLMs that naturally enables off-policy learning. We propose ReVal, a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification. ReVal naturally supports replay-buffer-based training, allowing efficient reuse of past trajectories. Experiments on standard mathematical reasoning benchmarks show that ReVal not only converges faster but also outperforms GRPO in final performance. On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO. These results suggest that value-based RL is a practical alternative to policy-based methods for LLM training.
@article{arxiv.2603.23355,
title = {Off-Policy Value-Based Reinforcement Learning for Large Language Models},
author = {Peng-Yuan Wang and Ziniu Li and Tian Xu and Bohan Yang and Tian-Shuo Liu and ChenYang Wang and Xiong-Hui Chen and Yi-Chen Li and Tianyun Yang and Congliang Chen and Yang Yu},
journal= {arXiv preprint arXiv:2603.23355},
year = {2026}
}