Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics
Abstract
Recurrent neural networks (RNNs) have gained a great deal of attention in solving sequential learning problems. The learning of long-term dependencies, however, remains challenging due to the problem of a vanishing or exploding hidden states gradient. By exploring further the recently established connections between RNNs and dynamical systems we propose a novel RNN architecture, which we call a Hamiltonian recurrent neural network (Hamiltonian RNN), based on a symplectic discretization of an appropriately chosen Hamiltonian system. The key benefit of this approach is that the corresponding RNN inherits the favorable long time properties of the Hamiltonian system, which in turn allows us to control the hidden states gradient with a hyperparameter of the Hamiltonian RNN architecture. This enables us to handle sequential learning problems with arbitrary sequence lengths, since for a range of values of this hyperparameter the gradient neither vanishes nor explodes. Additionally, we provide a heuristic for the optimal choice of the hyperparameter, which we use in our numerical simulations to illustrate that the Hamiltonian RNN is able to outperform other state-of-the-art RNNs without the need of computationally intensive hyperparameter optimization.
Cite
@article{arxiv.1911.05035,
title = {Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics},
author = {Konstantin Rusch and John W. Pearson and Konstantinos C. Zygalakis},
journal= {arXiv preprint arXiv:1911.05035},
year = {2020}
}
Comments
Reasons: 1. theoretical result of bounding the gradient dynamics is highly important when tackling the exploding gradient problem. However, we only proved the boundedness in one dimension and cannot generalize to the higher dimensional case, as the Hamiltonian argument is not valid in the general higher dimensional case. 2. The only medium strong performance on the widely used sMNIST problem