Short-term Memory of Deep RNN
Abstract
The extension of deep learning towards temporal data processing is gaining an increasing research interest. In this paper we investigate the properties of state dynamics developed in successive levels of deep recurrent neural networks (RNNs) in terms of short-term memory abilities. Our results reveal interesting insights that shed light on the nature of layering as a factor of RNN design. Noticeably, higher layers in a hierarchically organized RNN architecture results to be inherently biased towards longer memory spans even prior to training of the recurrent connections. Moreover, in the context of Reservoir Computing framework, our analysis also points out the benefit of a layered recurrent organization as an efficient approach to improve the memory skills of reservoir models.
Cite
@article{arxiv.1802.00748,
title = {Short-term Memory of Deep RNN},
author = {Claudio Gallicchio},
journal= {arXiv preprint arXiv:1802.00748},
year = {2018}
}
Comments
This is a pre-print (pre-review) version of the paper accepted for presentation at the 26th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), Bruges (Belgium), 25-27 April 2018