Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds

Masafumi Yamazaki; Akihiko Kasagi; Akihiro Tabuchi; Takumi Honda; Masahiro Miwa; Naoto Fukumoto; Tsuguchika Tabaru; Atsushi Ike; Kohta Nakashima

Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds

Machine Learning 2019-04-01 v1 Machine Learning

Authors: Masafumi Yamazaki , Akihiko Kasagi , Akihiro Tabuchi , Takumi Honda , Masahiro Miwa , Naoto Fukumoto , Tsuguchika Tabaru , Atsushi Ike , Kohta Nakashima

View on arXiv ↗ PDF ↗

Abstract

There has been a strong demand for algorithms that can execute machine learning as faster as possible and the speed of deep learning has accelerated by 30 times only in the past two years. Distributed deep learning using the large mini-batch is a key technology to address the demand and is a great challenge as it is difficult to achieve high scalability on large clusters without compromising accuracy. In this paper, we introduce optimization methods which we applied to this challenge. We achieved the training time of 74.7 seconds using 2,048 GPUs on ABCI cluster applying these methods. The training throughput is over 1.73 million images/sec and the top-1 validation accuracy is 75.08%.

Keywords

large mini-batch sgd distributed deep learning resnet-50 imagenet training acceleration optimization methods abci cluster

Cite

@article{arxiv.1903.12650,
  title  = {Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds},
  author = {Masafumi Yamazaki and Akihiko Kasagi and Akihiro Tabuchi and Takumi Honda and Masahiro Miwa and Naoto Fukumoto and Tsuguchika Tabaru and Atsushi Ike and Kohta Nakashima},
  journal= {arXiv preprint arXiv:1903.12650},
  year   = {2019}
}

Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds

Abstract

Keywords

Cite

Related papers