Recognizing less salient features is the key for model compression. However, it has not been investigated in the revolutionary attention mechanisms. In this work, we propose a novel normalization-based attention module (NAM), which suppresses less salient weights. It applies a weight sparsity penalty to the attention modules, thus, making them more computational efficient while retaining similar performance. A comparison with three other attention mechanisms on both Resnet and Mobilenet indicates that our method results in higher accuracy. Code for this paper can be publicly accessed at https://github.com/Christian-lyc/NAM.
@article{arxiv.2111.12419,
title = {NAM: Normalization-based Attention Module},
author = {Yichao Liu and Zongru Shao and Yueyang Teng and Nico Hoffmann},
journal= {arXiv preprint arXiv:2111.12419},
year = {2021}
}
Comments
3 pages, 2 figures, 2 tables, 2 tables in the appendix