Do RNN and LSTM have Long Memory?
Abstract
The LSTM network was proposed to overcome the difficulty in learning long-term dependence, and has made significant advancements in applications. With its success and drawbacks in mind, this paper raises the question - do RNN and LSTM have long memory? We answer it partially by proving that RNN and LSTM do not have long memory from a statistical perspective. A new definition for long memory networks is further introduced, and it requires the model weights to decay at a polynomial rate. To verify our theory, we convert RNN and LSTM into long memory networks by making a minimal modification, and their superiority is illustrated in modeling long-term dependence of various datasets.
Keywords
Cite
@article{arxiv.2006.03860,
title = {Do RNN and LSTM have Long Memory?},
author = {Jingyu Zhao and Feiqing Huang and Jia Lv and Yanjie Duan and Zhen Qin and Guodong Li and Guangjian Tian},
journal= {arXiv preprint arXiv:2006.03860},
year = {2020}
}
Comments
Accepted by ICML 2020. Added references, experiments and acknowledgements