Recent theory suggests that reward-model-first methods can be more sample-efficient than direct policy fitting when the reward function is statistically simpler than the induced policy. We propose DDO-RM, a finite-candidate decision-optimization method that converts reward scores into an explicit target distribution. Unlike PPO-based RLHF or DPO, DDO-RM performs a KL-regularized mirror-descent update to project the policy toward a reward-improved distribution over a candidate set. Preliminary experiments on Pythia-410M show that DDO-RM outperforms DPO in pair accuracy (0.52 to 0.56) and mean margin (0.13 to 0.53). Our framework provides a principled connection between reward learning and mirror-descent policy improvement.
@article{arxiv.2604.11119,
title = {DDO-RM: Distribution-Level Policy Improvement after Reward Learning},
author = {Tiantian Zhang and Jierui Zuo and Michael Chen and Wenping Wang},
journal= {arXiv preprint arXiv:2604.11119},
year = {2026}
}