Deep Neural Network Training without Multiplications

Tsuguo Mogami

Deep Neural Network Training without Multiplications

Machine Learning 2020-12-08 v1

Authors: Tsuguo Mogami

Abstract

Is multiplication really necessary for deep neural networks? Here we propose just adding two IEEE754 floating-point numbers with an integer-add instruction in place of a floating-point multiplication instruction. We show that ResNet can be trained using this operation with competitive classification accuracy. Our proposal did not require any methods to solve instability and decrease in accuracy, which is common in low-precision training. In some settings, we may obtain equal accuracy to the baseline FP32 result. This method will enable eliminating the multiplications in deep neural-network training and inference.

Keywords

neural network training mixed precision training deep learning

Cite

@article{arxiv.2012.03458,
  title  = {Deep Neural Network Training without Multiplications},
  author = {Tsuguo Mogami},
  journal= {arXiv preprint arXiv:2012.03458},
  year   = {2020}
}

Comments

6 pages, 1 figure, accepted as a workshop paper at NeurIPS2020

Related papers

View all related →