通过细粒度离策略指导在 RLVR 中回收探索

基于可验证奖励的强化学习 (RLVR) 已成为增强 Large Reasoning Models 复杂推理能力 的 powerful paradigm。然而，standard outcome-based supervision 存在 critical limitation，即惩罚由于 几个 missteps 而 fail 的 trajectories 与 completely erroneous ones 同样严重。这一 coarse feedback signal 导致 model 丢弃 valuable largely correct rollouts，leading to rollout diversity 的 degradation，进而 premature narrows exploration space。Process Reward Models 在 providing reliable step-wise verification for test-time scaling 中 demonstrated efficacy，naively 将 these signals 整合为 dense rewards proves ineffective。先前方法 attempt 引入 off-policy guided whole-trajectory replacement，often outside policy model's distribution，但 still fail to utilize 由 model 本身生成的 largely correct rollouts，from which do not effectively mitigate exploration space 的 narrowing。为解决这些问题，我们提出了 SCOPE (Step-wise Correction for On-Policy Exploration)，一个 novel framework，utilizes Process Reward Models 来 pinpoint first erroneous step in suboptimal rollouts，并 applies fine-grained, step-wise off-policy rectification。通过对 partially correct rollout 的 precise refinement，我们的方法 effectively salvages partially correct trajectories 并 increases diversity score by 13.5%，从而 sustain broad exploration space。Extensive experiments demonstrate that our approach establishes new state-of-the-art results，achieving average accuracy of 46.6% on math reasoning 并 exhibit robust generalization with 53.4% accuracy on out-of-distribution reasoning tasks。