Context-Free Transductions with Neural Stacks
Abstract
This paper analyzes the behavior of stack-augmented recurrent neural network (RNN) models. Due to the architectural similarity between stack RNNs and pushdown transducers, we train stack RNN models on a number of tasks, including string reversal, context-free language modelling, and cumulative XOR evaluation. Examining the behavior of our networks, we show that stack-augmented RNNs can discover intuitive stack-based strategies for solving our tasks. However, stack RNNs are more difficult to train than classical architectures such as LSTMs. Rather than employ stack-based strategies, more complex networks often find approximate solutions by using the stack as unstructured memory.
Cite
@article{arxiv.1809.02836,
title = {Context-Free Transductions with Neural Stacks},
author = {Yiding Hao and William Merrill and Dana Angluin and Robert Frank and Noah Amsel and Andrew Benz and Simon Mendelsohn},
journal= {arXiv preprint arXiv:1809.02836},
year = {2018}
}
Comments
To appear in the proceedings of the Analyzing and Interpreting Neural Networks for NLP workshop at EMNLP 2018