HGRN2: Gated Linear RNNs with State Expansion
Abstract
Hierarchically gated linear RNN (HGRN, \citealt{HGRN}) has demonstrated competitive training speed and performance in language modeling while offering efficient inference. However, the recurrent state size of HGRN remains relatively small, limiting its expressiveness. To address this issue, we introduce a simple outer product-based state expansion mechanism, which significantly enlarges the recurrent state size without introducing any additional parameters. This enhancement also provides a linear attention interpretation for HGRN2, enabling hardware-efficient training. Our extensive experiments verify the advantage of HGRN2 over HGRN consistently across different settings and competitive with other recurrent models.
Cite
@article{arxiv.2404.07904,
title = {HGRN2: Gated Linear RNNs with State Expansion},
author = {Zhen Qin and Songlin Yang and Weixuan Sun and Xuyang Shen and Dong Li and Weigao Sun and Yiran Zhong},
journal= {arXiv preprint arXiv:2404.07904},
year = {2024}
}
Comments
Accept to COLM 2024. Yiran Zhong is the corresponding author. Zhen Qin and Songlin Yang contributed equally to this work. The source code is available at https://github.com/OpenNLPLab/HGRN2