Reward Model Perspectives: Whose Opinions Do Reward Models Reward?

Elle

Reward Model Perspectives: Whose Opinions Do Reward Models Reward?

Computation and Language 2025-10-09 v1 Artificial Intelligence

Authors: Elle

Abstract

Reward models (RMs) are central to the alignment of language models (LMs). An RM often serves as a proxy for human preferences to guide downstream LM behavior. However, our understanding of RM behavior is limited. Our work (i) formalizes a framework for measuring the alignment of opinions captured by RMs, (ii) investigates the extent to which RMs demonstrate sociodemographic biases, and (iii) explores the effects of prompting to steer rewards towards the preferences of a target group. We study the subjective and diverse perspectives on controversial topics, which allows us to quantify RM perspectives in terms of their opinions, attitudes, and values. We show that RMs are poorly aligned with several demographic groups and can systematically reward harmful stereotypes, and steering alone is not enough to overcome these limitations. Our findings underscore the need for more careful consideration of RM behavior in model alignment during preference learning to prevent the propagation of unwanted social biases in the language technologies that we use.

Keywords

reinforcement learning from human feedback fairness in machine learning language modeling

Cite

@article{arxiv.2510.06391,
  title  = {Reward Model Perspectives: Whose Opinions Do Reward Models Reward?},
  author = {Elle},
  journal= {arXiv preprint arXiv:2510.06391},
  year   = {2025}
}

Comments

Published at EMNLP 2025 under the full author name "Elle"

Reward Model Perspectives: Whose Opinions Do Reward Models Reward?

Abstract

Keywords

Cite

Comments

Related papers