人类反馈强化学习的开放问题与根本局限
人工智能
2023-09-12 v2 计算与语言
机器学习
摘要
人类反馈强化学习（RLHF）是一种训练 AI 系统以与人类目标对齐的技术。RLHF 已成为用于微调最先进大语言模型（LLMs）的核心方法。尽管广受欢迎，公开系统地梳理其缺陷的工作却相对较少。在本文中，我们（1）综述 RLHF 及相关方法的开放问题与根本局限；（2）概述在实践中理解、改进和补充 RLHF 的技术；以及（3）提出审计与披露标准以改进对 RLHF 系统的社会监督。我们的工作强调 RLHF 的局限性，并凸显多方位开发更安全 AI 系统方法的重要性。
引用
@article{arxiv.2307.15217,
title = {Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback},
author = {Stephen Casper and Xander Davies and Claudia Shi and Thomas Krendl Gilbert and Jérémy Scheurer and Javier Rando and Rachel Freedman and Tomasz Korbak and David Lindner and Pedro Freire and Tony Wang and Samuel Marks and Charbel-Raphaël Segerie and Micah Carroll and Andi Peng and Phillip Christoffersen and Mehul Damani and Stewart Slocum and Usman Anwar and Anand Siththaranjan and Max Nadeau and Eric J. Michaud and Jacob Pfau and Dmitrii Krasheninnikov and Xin Chen and Lauro Langosco and Peter Hase and Erdem Bıyık and Anca Dragan and David Krueger and Dorsa Sadigh and Dylan Hadfield-Menell},
journal= {arXiv preprint arXiv:2307.15217},
year = {2023}
}