PaLM：利用Pathways扩展语言建模
计算与语言
2022-10-06 v5
摘要
大语言模型（LLM）已通过使用少样本学习在多种自然语言任务上展现出卓越性能，这大幅减少了使模型适配特定应用所需的任务专用训练样本数量。为进一步理解规模对少样本学习的影响，我们训练了一个5400亿参数、稠密激活的Transformer语言模型，称之为Pathways语言模型PaLM。我们使用Pathways（一种可在多个TPU Pod间实现高效训练的新型ML系统）在6144个TPU v4芯片上训练了PaLM。我们通过在数百个语言理解与生成基准上取得最先进的少样本学习效果，展示了规模化的持续收益。在其中的许多任务上，PaLM 540B取得了突破性性能，在多步推理任务套件上优于微调的最先进模型，并在最近发布的BIG-bench基准上优于人类平均表现。大量BIG-bench任务显示出因模型规模带来的不连续提升，即性能随着扩展到最大模型而急剧上升。PaLM在多语言任务和源代码生成方面也具备强大能力，我们在广泛基准上进行了展示。此外，我们对偏见与毒性进行了全面分析，并研究了训练数据记忆随模型规模的程度。最后，我们讨论了与大语言模型相关的伦理考量并探讨了潜在的缓解策略。
引用
@article{arxiv.2204.02311,
title = {PaLM: Scaling Language Modeling with Pathways},
author = {Aakanksha Chowdhery and Sharan Narang and Jacob Devlin and Maarten Bosma and Gaurav Mishra and Adam Roberts and Paul Barham and Hyung Won Chung and Charles Sutton and Sebastian Gehrmann and Parker Schuh and Kensen Shi and Sasha Tsvyashchenko and Joshua Maynez and Abhishek Rao and Parker Barnes and Yi Tay and Noam Shazeer and Vinodkumar Prabhakaran and Emily Reif and Nan Du and Ben Hutchinson and Reiner Pope and James Bradbury and Jacob Austin and Michael Isard and Guy Gur-Ari and Pengcheng Yin and Toju Duke and Anselm Levskaya and Sanjay Ghemawat and Sunipa Dev and Henryk Michalewski and Xavier Garcia and Vedant Misra and Kevin Robinson and Liam Fedus and Denny Zhou and Daphne Ippolito and David Luan and Hyeontaek Lim and Barret Zoph and Alexander Spiridonov and Ryan Sepassi and David Dohan and Shivani Agrawal and Mark Omernick and Andrew M. Dai and Thanumalayan Sankaranarayana Pillai and Marie Pellat and Aitor Lewkowycz and Erica Moreira and Rewon Child and Oleksandr Polozov and Katherine Lee and Zongwei Zhou and Xuezhi Wang and Brennan Saeta and Mark Diaz and Orhan Firat and Michele Catasta and Jason Wei and Kathy Meier-Hellstern and Douglas Eck and Jeff Dean and Slav Petrov and Noah Fiedel},
journal= {arXiv preprint arXiv:2204.02311},
year = {2022}
}