Training Language Models Using Target-Propagation
Computation and Language
2017-02-17 v1 Machine Learning
Neural and Evolutionary Computing
Abstract
While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making parallelization difficult) and from truncating gradient flow between distant time-steps. We investigate whether Target Propagation (TPROP) style approaches can address these shortcomings. Unfortunately, extensive experiments suggest that TPROP generally underperforms BPTT, and we end with an analysis of this phenomenon, and suggestions for future work.
Cite
@article{arxiv.1702.04770,
title = {Training Language Models Using Target-Propagation},
author = {Sam Wiseman and Sumit Chopra and Marc'Aurelio Ranzato and Arthur Szlam and Ruoyu Sun and Soumith Chintala and Nicolas Vasilache},
journal= {arXiv preprint arXiv:1702.04770},
year = {2017}
}