TAPS is a Topology-Aware intra-operator Parallelism strategy Searching algorithm that generates intra-operator parallelism strategies by considering both intra-node and inter-node bandwidth. Most of the existing auto-parallelism works use the communication volume as the communication cost directly when generating strategies, which we prove to be sub-optimal in multi-nodes cases. We design a topology-aware cost model for multi-node intra-operator parallelism strategy searching. Numerical experiments demonstrate that TAPS can generate strategies with up to 85% fewer communication costs, which outperform the latest baselines.
@article{arxiv.2301.04285,
title = {TAPS: Topology-Aware Intra-Operator Parallelism Strategy Searching Algorithm for Deep Neural Networks},
author = {Peng Liang and Hao Zheng and Teng Su and Linbo Qiao and Dongsheng Li},
journal= {arXiv preprint arXiv:2301.04285},
year = {2023}
}
Comments
11 pages, 6 figures. To be submitted to conference proceedings or a journal after modifications