Long-term series forecasting with Query Selector -- efficient model of sparse attention

Jacek Klimek; Jakub Klimek; Witold Kraskiewicz; Mateusz Topolewski

Long-term series forecasting with Query Selector -- efficient model of sparse attention

Machine Learning 2021-08-18 v2

Authors: Jacek Klimek , Jakub Klimek , Witold Kraskiewicz , Mateusz Topolewski

Abstract

Various modifications of TRANSFORMER were recently used to solve time-series forecasting problem. We propose Query Selector - an efficient, deterministic algorithm for sparse attention matrix. Experiments show it achieves state-of-the art results on ETT, Helpdesk and BPI'12 datasets.

Keywords

time series forecasting attention mechanism time series classification

Cite

@article{arxiv.2107.08687,
  title  = {Long-term series forecasting with Query Selector -- efficient model of sparse attention},
  author = {Jacek Klimek and Jakub Klimek and Witold Kraskiewicz and Mateusz Topolewski},
  journal= {arXiv preprint arXiv:2107.08687},
  year   = {2021}
}

Comments

10 pages

Related papers

View all related →

Computation and Language · Computer Science

Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers

Chao Lou, Zixia Jia, Zilong Zheng, Kewei Tu

2024-06-25

Computation and Language · Computer Science

Predicting Attention Sparsity in Transformers

Marcos Treviso, António Góis, Patrick Fernandes, Erick Fonseca +1

2022-04-22