Recurrent Dropout without Memory Loss
Computation and Language
2016-08-08 v2
Abstract
This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to \textit{forward} connections of feed-forward architectures or RNNs, we propose to drop neurons directly in \textit{recurrent} connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.
Cite
@article{arxiv.1603.05118,
title = {Recurrent Dropout without Memory Loss},
author = {Stanislau Semeniuta and Aliaksei Severyn and Erhardt Barth},
journal= {arXiv preprint arXiv:1603.05118},
year = {2016}
}