Equi-normalization of Neural Networks
Computer Vision and Pattern Recognition
2019-02-28 v1 Machine Learning
Abstract
Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and output weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the L2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch-and group-normalization on CIFAR-10 and ImageNet with a ResNet-18.
Cite
@article{arxiv.1902.10416,
title = {Equi-normalization of Neural Networks},
author = {Pierre Stock and Benjamin Graham and Rémi Gribonval and Hervé Jégou},
journal= {arXiv preprint arXiv:1902.10416},
year = {2019}
}
Comments
ICLR 2019 camera-ready