Higher Order Linear Transformer

Jean Mercat

Higher Order Linear Transformer

Machine Learning 2020-10-29 v1

Authors: Jean Mercat

Abstract

Following up on the linear transformer part of the article from Katharopoulos et al., that takes this idea from Shen et al., the trick that produces a linear complexity for the attention mechanism is re-used and extended to a second-order approximation of the softmax normalization.

Cite

@article{arxiv.2010.14816,
  title  = {Higher Order Linear Transformer},
  author = {Jean Mercat},
  journal= {arXiv preprint arXiv:2010.14816},
  year   = {2020}
}

Related papers

View all related →