DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
Computation and Language
2019-07-29 v2
Abstract
Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.
Cite
@article{arxiv.1907.11065,
title = {DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks},
author = {Lin Zehui and Pengfei Liu and Luyao Huang and Junkun Chen and Xipeng Qiu and Xuanjing Huang},
journal= {arXiv preprint arXiv:1907.11065},
year = {2019}
}