Task-Synchronized Recurrent Neural Networks
Abstract
Data are often sampled irregularly in time. Dealing with this using Recurrent Neural Networks (RNNs) traditionally involved ignoring the fact, feeding the time differences as additional inputs, or resampling the data. All these methods have their shortcomings. We propose an elegant straightforward alternative approach where instead the RNN is in effect resampled in time to match the time of the data or the task at hand. We use Echo State Network (ESN) and Gated Recurrent Unit (GRU) as the basis for our solution. Such RNNs can be seen as discretizations of continuous-time dynamical systems, which gives a solid theoretical ground to our approach. Our Task-Synchronized ESN (TSESN) and GRU (TSGRU) models allow for a direct model time setting and require no additional training, parameter tuning, or computation (solving differential equations or interpolating data) compared to their regular counterparts, thus retaining their original efficiency. We confirm empirically that our models can effectively compensate for the time-non-uniformity of the data and demonstrate that they compare favorably to data resampling, classical RNN methods, and alternative RNN models proposed to deal with time irregularities on several real-world nonuniform-time datasets. We open-source the code at https://github.com/oshapio/task-synchronized-RNNs .
Cite
@article{arxiv.2204.05192,
title = {Task-Synchronized Recurrent Neural Networks},
author = {Mantas Lukoševičius and Arnas Uselis},
journal= {arXiv preprint arXiv:2204.05192},
year = {2024}
}
Comments
The 1st version was written in May 2019 and double-blind reviewed for a prominent conference. A major update. We changed the name of the article and methods to an arguably more precise one, and because a very similar title has been published in the meantime. We've rewritten much of the text, connected to the current literature, redone some experiments, figures, discussion, published source code