Restructuring Batch Normalization to Accelerate CNN Training
Abstract
Batch Normalization (BN) has become a core design block of modern Convolutional Neural Networks (CNNs). A typical modern CNN has a large number of BN layers in its lean and deep architecture. BN requires mean and variance calculations over each mini-batch during training. Therefore, the existing memory access reduction techniques, such as fusing multiple CONV layers, are not effective for accelerating BN due to their inability to optimize mini-batch related calculations during training. To address this increasingly important problem, we propose to restructure BN layers by first splitting a BN layer into two sub-layers (fission) and then combining the first sub-layer with its preceding CONV layer and the second sub-layer with the following activation and CONV layers (fusion). The proposed solution can significantly reduce main-memory accesses while training the latest CNN models, and the experiments on a chip multiprocessor show that the proposed BN restructuring can improve the performance of DenseNet-121 by 25.7%.
Cite
@article{arxiv.1807.01702,
title = {Restructuring Batch Normalization to Accelerate CNN Training},
author = {Wonkyung Jung and Daejin Jung and and Byeongho Kim and Sunjung Lee and Wonjong Rhee and Jung Ho Ahn},
journal= {arXiv preprint arXiv:1807.01702},
year = {2019}
}
Comments
13 pages, 8 figures, to appear in SysML 2019, added ResNet-50 results