LaMDA：面向对话应用的语言模型
计算与语言
2022-02-11 v3 人工智能
摘要
我们提出 LaMDA：面向对话应用的语言模型（LaMDA: Language Models for Dialog Applications）。LaMDA 是一系列基于 Transformer 的专用对话神经语言模型，参数量最高达 137B，并在 1.56T 词的公开对话数据与网络文本上进行了预训练。虽然仅靠模型规模扩展可提升质量，但在安全性与事实依据方面改进较小。我们证明，利用标注数据微调并让模型能够查阅外部知识源，可针对安全性与事实依据这两大关键挑战带来显著改进。第一大挑战安全性，涉及确保模型响应符合一套人类价值观，例如防止有害建议与不公平偏见。我们使用基于一组示例性人类价值观的指标来量化安全性，并发现利用少量众包标注数据微调的 LaMDA 分类器对候选响应进行过滤，是提升模型安全性的一种可行途径。第二大挑战事实依据，涉及使模型能够查阅信息检索系统、语言翻译器与计算器等外部知识源。我们使用依据性（groundedness）指标量化事实性，并发现我们的方法使模型能够生成基于已知来源的响应，而非仅听起来合理的响应。最后，我们探索 LaMDA 在教育与内容推荐领域的应用，并分析其有用性与角色一致性。
引用
@article{arxiv.2201.08239,
title = {LaMDA: Language Models for Dialog Applications},
author = {Romal Thoppilan and Daniel De Freitas and Jamie Hall and Noam Shazeer and Apoorv Kulshreshtha and Heng-Tze Cheng and Alicia Jin and Taylor Bos and Leslie Baker and Yu Du and YaGuang Li and Hongrae Lee and Huaixiu Steven Zheng and Amin Ghafouri and Marcelo Menegali and Yanping Huang and Maxim Krikun and Dmitry Lepikhin and James Qin and Dehao Chen and Yuanzhong Xu and Zhifeng Chen and Adam Roberts and Maarten Bosma and Vincent Zhao and Yanqi Zhou and Chung-Ching Chang and Igor Krivokon and Will Rusch and Marc Pickett and Pranesh Srinivasan and Laichee Man and Kathleen Meier-Hellstern and Meredith Ringel Morris and Tulsee Doshi and Renelito Delos Santos and Toju Duke and Johnny Soraker and Ben Zevenbergen and Vinodkumar Prabhakaran and Mark Diaz and Ben Hutchinson and Kristen Olson and Alejandra Molina and Erin Hoffman-John and Josh Lee and Lora Aroyo and Ravi Rajakumar and Alena Butryna and Matthew Lamm and Viktoriya Kuzmina and Joe Fenton and Aaron Cohen and Rachel Bernstein and Ray Kurzweil and Blaise Aguera-Arcas and Claire Cui and Marian Croak and Ed Chi and Quoc Le},
journal= {arXiv preprint arXiv:2201.08239},
year = {2022}
}