扩展语言模型：训练 Gopher 的方法、分析与洞见
计算与语言
2022-01-24 v2 人工智能
摘要
语言建模通过利用大规模书面人类知识库来更好地预测和理解世界，为智能通信系统迈出一步。本文中，我们呈现基于 Transformer 的语言模型在广泛模型规模上的性能分析——从数千万参数的模型到名为 Gopher 的 2800 亿参数模型。这些模型在 152 项多样任务上评估，在多数上取得最先进性能。规模带来的收益在阅读理解、事实核查和有毒语言识别等领域最大，但逻辑与数学推理受益较少。我们提供对训练数据集与模型行为的整体分析，涵盖模型规模与偏见和有毒性的交集。最后我们讨论语言模型在 AI 安全及缓解下游危害方面的应用。
引用
@article{arxiv.2112.11446,
title = {Scaling Language Models: Methods, Analysis & Insights from Training Gopher},
author = {Jack W. Rae and Sebastian Borgeaud and Trevor Cai and Katie Millican and Jordan Hoffmann and Francis Song and John Aslanides and Sarah Henderson and Roman Ring and Susannah Young and Eliza Rutherford and Tom Hennigan and Jacob Menick and Albin Cassirer and Richard Powell and George van den Driessche and Lisa Anne Hendricks and Maribeth Rauh and Po-Sen Huang and Amelia Glaese and Johannes Welbl and Sumanth Dathathri and Saffron Huang and Jonathan Uesato and John Mellor and Irina Higgins and Antonia Creswell and Nat McAleese and Amy Wu and Erich Elsen and Siddhant Jayakumar and Elena Buchatskaya and David Budden and Esme Sutherland and Karen Simonyan and Michela Paganini and Laurent Sifre and Lena Martens and Xiang Lorraine Li and Adhiguna Kuncoro and Aida Nematzadeh and Elena Gribovskaya and Domenic Donato and Angeliki Lazaridou and Arthur Mensch and Jean-Baptiste Lespiau and Maria Tsimpoukelli and Nikolai Grigorev and Doug Fritz and Thibault Sottiaux and Mantas Pajarskas and Toby Pohlen and Zhitao Gong and Daniel Toyama and Cyprien de Masson d'Autume and Yujia Li and Tayfun Terzi and Vladimir Mikulik and Igor Babuschkin and Aidan Clark and Diego de Las Casas and Aurelia Guy and Chris Jones and James Bradbury and Matthew Johnson and Blake Hechtman and Laura Weidinger and Iason Gabriel and William Isaac and Ed Lockhart and Simon Osindero and Laura Rimell and Chris Dyer and Oriol Vinyals and Kareem Ayoub and Jeff Stanway and Lorrayne Bennett and Demis Hassabis and Koray Kavukcuoglu and Geoffrey Irving},
journal= {arXiv preprint arXiv:2112.11446},
year = {2022}
}
备注
120 pages