English

RLHF and IIA: Perverse Incentives

Machine Learning 2024-02-02 v3 Artificial Intelligence Computation and Language
Authors: Wanqiao Xu , Shi Dong , Xiuyuan Lu , Grace Lam , Zheng Wen , Benjamin Van Roy
View on arXiv ↗ PDF ↗

Abstract

Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA). The perverse incentives induced by IIA hinder innovations on query formats and learning algorithms.

Keywords

reinforcement learning from human feedback

Cite

@article{arxiv.2312.01057,
  title  = {RLHF and IIA: Perverse Incentives},
  author = {Wanqiao Xu and Shi Dong and Xiuyuan Lu and Grace Lam and Zheng Wen and Benjamin Van Roy},
  journal= {arXiv preprint arXiv:2312.01057},
  year   = {2024}
}

Related papers

View all related →
Artificial Intelligence · Computer Science
Secrets of RLHF in Large Language Models Part II: Reward Modeling
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu +23
2024-01-15
Machine Learning · Computer Science
Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL
Nyal Patel, Matthieu Bou, Arjun Jagota, Satyapriya Krishna +1
2026-01-21
Computers and Society · Computer Science
The History and Risks of Reinforcement Learning and Human Feedback
Nathan Lambert, Thomas Krendl Gilbert, Tom Zick
2023-11-29
Machine Learning · Computer Science
Influencing Humans to Conform to Preference Models for RLHF
Stephane Hatgis-Kessell, W. Bradley Knox, Serena Booth, Peter Stone
2026-04-14
Machine Learning · Computer Science
A Survey of Reinforcement Learning from Human Feedback
Timo Kaufmann, Paul Weng, Viktor Bengs, Eyke Hüllermeier
2025-12-30
Machine Learning · Computer Science
Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta +1
2024-08-20
Machine Learning · Statistics
Reinforcement Learning from Human Feedback: A Statistical Perspective
Pangpang Liu, Chengchun Shi, Will Wei Sun
2026-04-06
Computation and Language · Computer Science
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard +7
2024-09-04
Computation and Language · Computer Science
Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng +2
2024-03-15
Machine Learning · Computer Science
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit +4
2024-04-17
Machine Learning · Computer Science
Provable Multi-Party Reinforcement Learning with Diverse Human Feedback
Huiying Zhong, Zhun Deng, Weijie J. Su, Zhiwei Steven Wu +1
2024-03-11
Computation and Language · Computer Science
Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
Jared Joselowitz, Ritam Majumdar, Arjun Jagota, Matthieu Bou +3
2025-10-07
Machine Learning · Statistics
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
Kai Ye, Hongyi Zhou, Jin Zhu, Francesco Quinzan +1
2026-02-11
Artificial Intelligence · Computer Science
Understanding Impact of Human Feedback via Influence Functions
Taywon Min, Haeone Lee, Yongchan Kwon, Kimin Lee
2025-09-03
Machine Learning · Computer Science
Exploring and Addressing Reward Confusion in Offline Preference Learning
Xin Chen, Sam Toyer, Florian Shkurti
2024-10-16
Machine Learning · Computer Science
Explainable reinforcement learning from human feedback to improve alignment
Shicheng Liu, Siyuan Xu, Wenjie Qiu, Hangfan Zhang +1
2025-12-17
Computation and Language · Computer Science
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell +27
2022-04-13
Artificial Intelligence · Computer Science
IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking
Mohammad Beigi, Ming Jin, Junshan Zhang, Jiaxin Zhang +2
2026-02-24
Machine Learning · Computer Science
Is RLHF More Difficult than Standard RL?
Yuanhao Wang, Qinghua Liu, Chi Jin
2023-11-07
Artificial Intelligence · Computer Science
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert +28
2023-09-12
Machine Learning · Computer Science
Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback
Jan Kompatscher, Danqing Shi, Giovanna Varni, Tino Weinkauf +1
2025-12-01
Artificial Intelligence · Computer Science
Mitigating Cognitive Bias in RLHF by Altering Rationality
Tiffany Horter, Andrew Markham, Niki Trigoni, Serena Booth
2026-05-11
Machine Learning · Statistics
Dual Active Learning for Reinforcement Learning from Human Feedback
Pangpang Liu, Chengchun Shi, Will Wei Sun
2025-01-03
Machine Learning · Computer Science
Contrastive Preference Learning: Learning from Human Feedback without RL
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn +3
2024-05-01
Machine Learning · Computer Science
A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback
Kihyun Kim, Jiawei Zhang, Asuman Ozdaglar, Pablo A. Parrilo
2024-10-16
R2 v1 2026-06-28T13:39:04.282Z