RecurrentGemma：超越 Transformer 的高效开放语言模型
机器学习
2024-08-29 v2 人工智能
计算与语言
摘要
我们介绍了 RecurrentGemma，一系列使用 Google 全新 Griffin 架构的开放语言模型。Griffin 将线性递归与局部注意力相结合，在语言任务上取得了卓越的性能。它具有固定大小的状态，从而减少了内存使用并能在长序列上实现高效推理。我们提供了包含 2B 和 9B 参数的两种规模的模型，并为两者提供了预训练和指令微调的变体。尽管在更少的 token 上进行训练，我们的模型仍取得了与同等规模的 Gemma 基线相当的性能。
引用
@article{arxiv.2404.07839,
title = {RecurrentGemma: Moving Past Transformers for Efficient Open Language Models},
author = {Aleksandar Botev and Soham De and Samuel L Smith and Anushan Fernando and George-Cristian Muraru and Ruba Haroun and Leonard Berrada and Razvan Pascanu and Pier Giuseppe Sessa and Robert Dadashi and Léonard Hussenot and Johan Ferret and Sertan Girgin and Olivier Bachem and Alek Andreev and Kathleen Kenealy and Thomas Mesnard and Cassidy Hardin and Surya Bhupatiraju and Shreya Pathak and Laurent Sifre and Morgane Rivière and Mihir Sanjay Kale and Juliette Love and Pouya Tafti and Armand Joulin and Noah Fiedel and Evan Senter and Yutian Chen and Srivatsan Srinivasan and Guillaume Desjardins and David Budden and Arnaud Doucet and Sharad Vikram and Adam Paszke and Trevor Gale and Sebastian Borgeaud and Charlie Chen and Andy Brock and Antonia Paterson and Jenny Brennan and Meg Risdal and Raj Gundluru and Nesh Devanathan and Paul Mooney and Nilay Chauhan and Phil Culliton and Luiz Gustavo Martins and Elisa Bandy and David Huntsperger and Glenn Cameron and Arthur Zucker and Tris Warkentin and Ludovic Peran and Minh Giang and Zoubin Ghahramani and Clément Farabet and Koray Kavukcuoglu and Demis Hassabis and Raia Hadsell and Yee Whye Teh and Nando de Frietas},
journal= {arXiv preprint arXiv:2404.07839},
year = {2024}
}