Population-based Global Optimisation Methods for Learning Long-term Dependencies with RNNs
Abstract
Despite recent innovations in network architectures and loss functions, training RNNs to learn long-term dependencies remains difficult due to challenges with gradient-based optimisation methods. Inspired by the success of Deep Neuroevolution in reinforcement learning (Such et al. 2017), we explore the use of gradient-free population-based global optimisation (PBO) techniques -- training RNNs to capture long-term dependencies in time-series data. Testing evolution strategies (ES) and particle swarm optimisation (PSO) on an application in volatility forecasting, we demonstrate that PBO methods lead to performance improvements in general, with ES exhibiting the most consistent results across a variety of architectures.
Cite
@article{arxiv.1905.09691,
title = {Population-based Global Optimisation Methods for Learning Long-term Dependencies with RNNs},
author = {Bryan Lim and Stefan Zohren and Stephen Roberts},
journal= {arXiv preprint arXiv:1905.09691},
year = {2019}
}
Comments
To appear at ICML 2019 Time Series Workshop