Batch Normalization in Quantized Networks
Machine Learning
2020-04-30 v1 Machine Learning
Abstract
Implementation of quantized neural networks on computing hardware leads to considerable speed up and memory saving. However, quantized deep networks are difficult to train and batch~normalization (BatchNorm) layer plays an important role in training full-precision and quantized networks. Most studies on BatchNorm are focused on full-precision networks, and there is little research in understanding BatchNorm affect in quantized training which we address here. We show BatchNorm avoids gradient explosion which is counter-intuitive and recently observed in numerical experiments by other researchers.
Cite
@article{arxiv.2004.14214,
title = {Batch Normalization in Quantized Networks},
author = {Eyyüb Sari and Vahid Partovi Nia},
journal= {arXiv preprint arXiv:2004.14214},
year = {2020}
}