Recent work has explored reduced numerical precision for parameters, activations, and gradients during neural network training as a way to reduce the computational cost of training (Na & Mukhopadhyay, 2016) (Courbariaux et al., 2014). We present a novel dynamic precision scaling (DPS) scheme. Using stochastic fixed-point rounding, a quantization-error based scaling scheme, and dynamic bit-widths during training, we achieve 98.8% test accuracy on the MNIST dataset using an average bit-width of just 16 bits for weights and 14 bits for activations, compared to the standard 32-bit floating point values used in deep learning frameworks.
@article{arxiv.1801.08621,
title = {Quantization Error as a Metric for Dynamic Precision Scaling in Neural Net Training},
author = {Ian Taras and Dylan Malone Stuart},
journal= {arXiv preprint arXiv:1801.08621},
year = {2019}
}