IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models
Abstract
Generative Reward Models (GRMs) have demonstrated strong performance in reward modeling, due to their interpretability and potential for refinement through reinforcement learning (RL). However, widely used pairwise GRMs create a computational bottleneck in reinforcement learning from human feedback (RLHF), when calibrating or aggregating preference signals over n candidates, often incurring O(n^2) pairwise judgments. To address this issue, we propose Intergroup Relative Preference Modeling (IRPM), an RL-based method that extends the Bradley--Terry preference-learning paradigm via intergroup comparisons to train pointwise GRMs from pairwise preference data. IRPM derives pointwise reward for each response by contrasting groups of chosen vs. rejected samples, enabling pointwise scores comparable across candidate sets and O(n) reward evaluation for a variable number of candidates during RL training, while preserving interpretability and scalability. Experiments show that IRPM achieves state-of-the-art performance among pointwise GRMs on RM-Bench, JudgeBench and RewardBench, and approaches the performance of leading pairwise GRMs. In addition, IRPM achieves substantial gains in post-training evaluations, demonstrating its effectiveness.
Cite
@article{arxiv.2601.00677,
title = {IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models},
author = {Haonan Song and Qingchen Xie and Huan Zhu and Feng Xiao and Luxi Xing and Liu Kang and Fuzhen Li and Zhiyong Zheng and Feng Jiang and Ziheng Li and Kun Yan and Qingyi Si and Yanghua Xiao and Hongcheng Guo and Fan Yang},
journal= {arXiv preprint arXiv:2601.00677},
year = {2026}
}
Comments
Comments: Updated title for clarity; improved theoretical derivations; added experiments at additional parameter scales and more ablations; added experimental details in the appendix; updated author list (added five co-authors) to reflect contributions to experiments and writing