Counting in Language with RNNs
Machine Learning
2018-11-01 v2 Neural and Evolutionary Computing
Machine Learning
Abstract
In this paper we examine a possible reason for the LSTM outperforming the GRU on language modeling and more specifically machine translation. We hypothesize that this has to do with counting. This is a consistent theme across the literature of long term dependence, counting, and language modeling for RNNs. Using the simplified forms of language -- Context-Free and Context-Sensitive Languages -- we show how exactly the LSTM performs its counting based on their cell states during inference and why the GRU cannot perform as well.
Cite
@article{arxiv.1810.12411,
title = {Counting in Language with RNNs},
author = {Heng xin Fun and Sergiy V Bokhnyak and Francesco Saverio Zuppichini},
journal= {arXiv preprint arXiv:1810.12411},
year = {2018}
}
Comments
Withdrawing due to lack of key acknowledgements and follow up work