The Hierarchical Attention Network (HAN) has made great strides, but it suffers a major limitation: at level 1, each sentence is encoded in complete isolation. In this work, we propose and compare several modifications of HAN in which the sentence encoder is able to make context-aware attentional decisions (CAHAN). Furthermore, we propose a bidirectional document encoder that processes the document forwards and backwards, using the preceding and following sentences as context. Experiments on three large-scale sentiment and topic classification datasets show that the bidirectional version of CAHAN outperforms HAN everywhere, with only a modest increase in computation time. While results are promising, we expect the superiority of CAHAN to be even more evident on tasks requiring a deeper understanding of the input documents, such as abstractive summarization. Code is publicly available.
@article{arxiv.1908.06006,
title = {Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding},
author = {Jean-Baptiste Remy and Antoine Jean-Pierre Tixier and Michalis Vazirgiannis},
journal= {arXiv preprint arXiv:1908.06006},
year = {2019}
}