A$^3$: Accelerating Attention Mechanisms in Neural Networks with Approximation
Abstract
With the increasing computational demands of neural networks, many hardware accelerators for the neural networks have been proposed. Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network primitive that enables neural networks to retrieve most relevant information from a knowledge-base, external memory, or past states. The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time. We observe today's practice of implementing this mechanism using matrix-vector multiplication is suboptimal as the attention mechanism is semantically a content-based search where a large portion of computations ends up not being used. Based on this observation, we design and architect A3, which accelerates attention mechanisms in neural networks with algorithmic approximation and hardware specialization. Our proposed accelerator achieves multiple orders of magnitude improvement in energy efficiency (performance/watt) as well as substantial speedup over the state-of-the-art conventional hardware.
Cite
@article{arxiv.2002.10941,
title = {A$^3$: Accelerating Attention Mechanisms in Neural Networks with Approximation},
author = {Tae Jun Ham and Sung Jun Jung and Seonghak Kim and Young H. Oh and Yeonhong Park and Yoonho Song and Jung-Hun Park and Sanghee Lee and Kyoung Park and Jae W. Lee and Deog-Kyoon Jeong},
journal= {arXiv preprint arXiv:2002.10941},
year = {2020}
}
Comments
To be published in 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)