大语言模型的道德自我修正能力
计算与语言
2023-02-21 v2
摘要
我们检验如下假设：经人类反馈强化学习（RLHF）训练的语言模型若被指示，则具备“道德自我修正”——即避免产生有害输出——的能力。我们在三个不同实验中找到支持该假设的强力证据，各实验揭示了道德自我修正的不同侧面。我们发现，道德自我修正能力涌现于220亿（22B）模型参数，且通常随模型规模增大与RLHF训练而提升。我们认为，在此规模下，语言模型获得可用于道德自我修正的两种能力：（1）遵循指令；（2）学习刻板印象、偏见与歧视等复杂的危害性规范概念。因此，它们可遵循指令避免某些道德上有害的输出。我们相信，我们的结果对于训练语言模型遵守伦理原则的能力而言，构成了审慎乐观的理由。
引用
@article{arxiv.2302.07459,
title = {The Capacity for Moral Self-Correction in Large Language Models},
author = {Deep Ganguli and Amanda Askell and Nicholas Schiefer and Thomas I. Liao and Kamilė Lukošiūtė and Anna Chen and Anna Goldie and Azalia Mirhoseini and Catherine Olsson and Danny Hernandez and Dawn Drain and Dustin Li and Eli Tran-Johnson and Ethan Perez and Jackson Kernion and Jamie Kerr and Jared Mueller and Joshua Landau and Kamal Ndousse and Karina Nguyen and Liane Lovitt and Michael Sellitto and Nelson Elhage and Noemi Mercado and Nova DasSarma and Oliver Rausch and Robert Lasenby and Robin Larson and Sam Ringer and Sandipan Kundu and Saurav Kadavath and Scott Johnston and Shauna Kravec and Sheer El Showk and Tamera Lanham and Timothy Telleen-Lawton and Tom Henighan and Tristan Hume and Yuntao Bai and Zac Hatfield-Dodds and Ben Mann and Dario Amodei and Nicholas Joseph and Sam McCandlish and Tom Brown and Christopher Olah and Jack Clark and Samuel R. Bowman and Jared Kaplan},
journal= {arXiv preprint arXiv:2302.07459},
year = {2023}
}