Several efficient distributed algorithms have been developed for matrix-matrix multiplication: the 3D algorithm, the 2D SUMMA algorithm, and the 2.5D algorithm. Each of these algorithms was independently conceived and they trade-off memory needed per node and the inter-node data communication volume. The convolutional neural network (CNN) computation may be viewed as a generalization of matrix-multiplication combined with neighborhood stencil computations. We develop communication-efficient distributed-memory algorithms for CNNs that are analogous to the 2D/2.5D/3D algorithms for matrix-matrix multiplication.
@article{arxiv.2105.13480,
title = {Efficient distributed algorithms for Convolutional Neural Networks},
author = {Rui Li and Yufan Xu and Aravind Sukumaran-Rajam and Atanas Rountev and P Sadayappan},
journal= {arXiv preprint arXiv:2105.13480},
year = {2021}
}
Comments
Proceedings of the 33rd ACM Symposium on Parallelism in Algorithms and Architectures (SPAA '21), July 6--8, 2021, Virtual Event, USA