Policy optimization for large language models often suffers from sparse reward signals in multi-step reasoning tasks. Critic-free methods like GRPO assign a single normalized outcome reward to all tokens, providing limited guidance for intermediate reasoning . While Process Reward Models (PRMs) offer dense feedback, they risk premature collapse when used alone, as early low-reward tokens can drive policies toward truncated outputs. We introduce Process Relative Policy Optimization (PRPO), which combines outcome reliability with process-level guidance in a critic-free framework. PRPO segments reasoning sequences based on semantic clues, normalizes PRM scores into token-level advantages, and aligns their distribution with outcome advantages through location-parameter shift. On MATH500, PRPO improves Qwen2.5-Math-1.5B accuracy from 61.2% to 64.4% over GRPO using only eight rollouts and no value network, demonstrating efficient fine-grained credit assignment within critic-free optimization. Code is available at: https://github.com/SchumiDing/srpocode
@article{arxiv.2601.07182,
title = {PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization},
author = {Ruiyi Ding and Yongxuan Lv and Xianhui Meng and Jiahe Song and Chao Wang and Chen Jiang and Yuan Cheng},
journal= {arXiv preprint arXiv:2601.07182},
year = {2026}
}
Comments
8 pages, 2 figures Code is available at: https://github.com/SchumiDing/srpocode