We propose the Taylorformer for random processes such as time series. Its two key components are: 1) the LocalTaylor wrapper which adapts Taylor approximations (used in dynamical systems) for use in neural network-based probabilistic models, and 2) the MHA-X attention block which makes predictions in a way inspired by how Gaussian Processes' mean predictions are linear smoothings of contextual data. Taylorformer outperforms the state-of-the-art in terms of log-likelihood on 5/6 classic Neural Process tasks such as meta-learning 1D functions, and has at least a 14\% MSE improvement on forecasting tasks, including electricity, oil temperatures and exchange rates. Taylorformer approximates a consistent stochastic process and provides uncertainty-aware predictions. Our code is provided in the supplementary material.
@article{arxiv.2305.19141,
title = {Taylorformer: Probabilistic Modelling for Random Processes including Time Series},
author = {Omer Nivron and Raghul Parthipan and Damon J. Wischik},
journal= {arXiv preprint arXiv:2305.19141},
year = {2024}
}
Comments
Presented at ICML 2023, New Frontiers in Learning, Control, and Dynamical Systems Workshop