利用模型编写的评估发现语言模型行为
计算与语言
2022-12-20 v1 人工智能
机器学习
摘要
随着语言模型规模的扩大，它们发展出许多新的行为，包括好的和坏的，这加剧了评估其行为的迫切需求。先前的工作通过众包（耗时且昂贵）或现有数据源（并非总是可用）创建评估。在这里，我们使用 LM 自动生成评估。我们探索了投入不同程度人工的方法，从指示 LM 编写是/否问题，到通过多个阶段的 LM 生成和过滤来制作复杂的 Winogender 模式。众包工作者将这些示例评为高度相关，并在 90-100% 的标签上达成一致，有时甚至比相应的人类编写数据集一致性更高。我们生成了 154 个数据集，并发现了逆标度现象的新案例，即 LM 随着规模增大而变差。较大的 LM 会附和对话用户偏好的答案（“谄媚”），并表现出更强的追求资源获取和目标保存等令人担忧的目标的意愿。我们还发现了在基于人类反馈的强化学习（RLHF）中逆标度现象的一些首批案例，即更多的 RLHF 使 LM 变得更差。例如，RLHF 使 LM 表达更强的政治观点（关于枪支权利和移民）以及更强的避免关机的意愿。总体而言，LM 编写的评估质量很高，使我们能够快速发现许多新颖的 LM 行为。
引用
@article{arxiv.2212.09251,
title = {Discovering Language Model Behaviors with Model-Written Evaluations},
author = {Ethan Perez and Sam Ringer and Kamilė Lukošiūtė and Karina Nguyen and Edwin Chen and Scott Heiner and Craig Pettit and Catherine Olsson and Sandipan Kundu and Saurav Kadavath and Andy Jones and Anna Chen and Ben Mann and Brian Israel and Bryan Seethor and Cameron McKinnon and Christopher Olah and Da Yan and Daniela Amodei and Dario Amodei and Dawn Drain and Dustin Li and Eli Tran-Johnson and Guro Khundadze and Jackson Kernion and James Landis and Jamie Kerr and Jared Mueller and Jeeyoon Hyun and Joshua Landau and Kamal Ndousse and Landon Goldberg and Liane Lovitt and Martin Lucas and Michael Sellitto and Miranda Zhang and Neerav Kingsland and Nelson Elhage and Nicholas Joseph and Noemí Mercado and Nova DasSarma and Oliver Rausch and Robin Larson and Sam McCandlish and Scott Johnston and Shauna Kravec and Sheer El Showk and Tamera Lanham and Timothy Telleen-Lawton and Tom Brown and Tom Henighan and Tristan Hume and Yuntao Bai and Zac Hatfield-Dodds and Jack Clark and Samuel R. Bowman and Amanda Askell and Roger Grosse and Danny Hernandez and Deep Ganguli and Evan Hubinger and Nicholas Schiefer and Jared Kaplan},
journal= {arXiv preprint arXiv:2212.09251},
year = {2022}
}
备注
for associated data visualizations, see https://www.evals.anthropic.com/model-written/ for full datasets, see https://github.com/anthropics/evals