使用 $\texttt{t5x}$ 与 $\texttt{seqio}$ 扩展模型与数据规模
机器学习
2022-04-01 v1 计算与语言
摘要
近期基于神经网络的语言模型从训练数据集规模的扩大以及模型自身参数量的增加中获益良多。由于诸多因素，扩展可能较为复杂，包括在超级计算机集群（如 TPU）上分配计算、防止数据输入时的瓶颈以及确保可复现的结果。在本工作中，我们提出两个缓解这些问题的软件库： 简化了大规模构建和训练大语言模型的过程，同时保持易用性； 提供了基于任务的 API，用于简单创建快速且可复现的训练数据与评估流水线。这些开源库已用于在具有数太字节训练数据的数据集上训练具有数千亿参数的模型。连同这些库，我们发布了类 T5 的编码器-解码器模型以及类 GPT 的仅解码器架构的配置与说明。 与 均为开源软件，分别可在 https://github.com/google-research/t5x 与 https://github.com/google/seqio 获取。
引用
@article{arxiv.2203.17189,
title = {Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$},
author = {Adam Roberts and Hyung Won Chung and Anselm Levskaya and Gaurav Mishra and James Bradbury and Daniel Andor and Sharan Narang and Brian Lester and Colin Gaffney and Afroz Mohiuddin and Curtis Hawthorne and Aitor Lewkowycz and Alex Salcianu and Marc van Zee and Jacob Austin and Sebastian Goodman and Livio Baldini Soares and Haitang Hu and Sasha Tsvyashchenko and Aakanksha Chowdhery and Jasmijn Bastings and Jannis Bulian and Xavier Garcia and Jianmo Ni and Andrew Chen and Kathleen Kenealy and Jonathan H. Clark and Stephan Lee and Dan Garrette and James Lee-Thorp and Colin Raffel and Noam Shazeer and Marvin Ritter and Maarten Bosma and Alexandre Passos and Jeremy Maitin-Shepard and Noah Fiedel and Mark Omernick and Brennan Saeta and Ryan Sepassi and Alexander Spiridonov and Joshua Newlan and Andrea Gesmundo},
journal= {arXiv preprint arXiv:2203.17189},
year = {2022}
}