Depth-Gated LSTM
Neural and Evolutionary Computing
2015-08-26 v4 Computation and Language
Abstract
In this short note, we present an extension of long short-term memory (LSTM) neural networks to using a depth gate to connect memory cells of adjacent layers. Doing so introduces a linear dependence between lower and upper layer recurrent units. Importantly, the linear dependence is gated through a gating function, which we call depth gate. This gate is a function of the lower layer memory cell, the input to and the past memory cell of this layer. We conducted experiments and verified that this new architecture of LSTMs was able to improve machine translation and language modeling performances.
Cite
@article{arxiv.1508.03790,
title = {Depth-Gated LSTM},
author = {Kaisheng Yao and Trevor Cohn and Katerina Vylomova and Kevin Duh and Chris Dyer},
journal= {arXiv preprint arXiv:1508.03790},
year = {2015}
}
Comments
Content presented in 2015 Jelinek Summer Workshop on Speech and Language Technology on August 14th 2015