HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization
Abstract
To capture the semantic graph structure from raw text, most existing summarization approaches are built on GNNs with a pre-trained model. However, these methods suffer from cumbersome procedures and inefficient computations for long-text documents. To mitigate these issues, this paper proposes HETFORMER, a Transformer-based pre-trained model with multi-granularity sparse attentions for long-text extractive summarization. Specifically, we model different types of semantic nodes in raw text as a potential heterogeneous graph and directly learn heterogeneous relationships (edges) among nodes by Transformer. Extensive experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1 while using less memory and fewer parameters.
Cite
@article{arxiv.2110.06388,
title = {HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization},
author = {Ye Liu and Jian-Guo Zhang and Yao Wan and Congying Xia and Lifang He and Philip S. Yu},
journal= {arXiv preprint arXiv:2110.06388},
year = {2021}
}
Comments
EMNLP 2021 (short paper)