Multi-Scale Self-Attention for Text Classification

Qipeng Guo; Xipeng Qiu; Pengfei Liu; Xiangyang Xue; Zheng Zhang

Multi-Scale Self-Attention for Text Classification

Computation and Language 2019-12-03 v1 Machine Learning

Authors: Qipeng Guo , Xipeng Qiu , Pengfei Liu , Xiangyang Xue , Zheng Zhang

Abstract

In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi-Scale Transformer which uses multi-scale multi-head self-attention to capture features from different scales. Based on the linguistic perspective and the analysis of pre-trained Transformer (BERT) on a huge corpus, we further design a strategy to control the scale distribution for each layer. Results of three different kinds of tasks (21 datasets) show our Multi-Scale Transformer outperforms the standard Transformer consistently and significantly on small and moderate size datasets.

Keywords

attention mechanism model transformation transformer

Cite

@article{arxiv.1912.00544,
  title  = {Multi-Scale Self-Attention for Text Classification},
  author = {Qipeng Guo and Xipeng Qiu and Pengfei Liu and Xiangyang Xue and Zheng Zhang},
  journal= {arXiv preprint arXiv:1912.00544},
  year   = {2019}
}

Comments

Accepted in AAAI2020

Multi-Scale Self-Attention for Text Classification

Abstract

Keywords

Cite

Comments

Related papers