We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.
@article{arxiv.1911.02972,
title = {Blockwise Self-Attention for Long Document Understanding},
author = {Jiezhong Qiu and Hao Ma and Omer Levy and Scott Wen-tau Yih and Sinong Wang and Jie Tang},
journal= {arXiv preprint arXiv:1911.02972},
year = {2020}
}
Comments
Accepted at Findings of EMNLP'20 and SustaiNLP 2020 at EMNLP'20, 12 pages