Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.
@article{arxiv.2601.08521,
title = {Your Group-Relative Advantage Is Biased},
author = {Fengkai Yang and Zherui Chen and Xiaohan Wang and Xiaodong Lu and Jiajun Chai and Guojun Yin and Wei Lin and Shuai Ma and Fuzhen Zhuang and Deqing Wang and Yaodong Yang and Jianxin Li and Yikun Ban},
journal= {arXiv preprint arXiv:2601.08521},
year = {2026}
}