This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Output Gate (IOG). IOG has an extremely simple structure, and thus, can be easily combined with any RNN language models. Our experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG consistently boosts the performance of several different types of current topline RNN language models.
Cite
@article{arxiv.1709.08907,
title = {Input-to-Output Gate to Improve RNN Language Models},
author = {Sho Takase and Jun Suzuki and Masaaki Nagata},
journal= {arXiv preprint arXiv:1709.08907},
year = {2017}
}