Binary neural networks improve computationally efficiency of deep models with a large margin. However, there is still a performance gap between a successful full-precision training and binary training. We bring some insights about why this accuracy drop exists and call for a better understanding of binary network geometry. We start with analyzing full-precision neural networks with ReLU activation and compare it with its binarized version. This comparison suggests to initialize networks with random bias, a counter-intuitive remedy.
@article{arxiv.1909.13446,
title = {Random Bias Initialization Improves Quantized Training},
author = {Xinlin Li and Vahid Partovi Nia},
journal= {arXiv preprint arXiv:1909.13446},
year = {2020}
}