Fire-Flyer AI-HPC：面向深度学习的低成本软件硬件协同设计
分布式、并行与集群计算
2024-09-04 v2 人工智能
摘要
深度学习（DL）和大型语言模型（LLM）的快速发展呈指数级增长，对计算功率和带宽的需求日益提升。结合更快计算芯片和互连器件的高成本，显著推高了高性能计算（HPC）的建设成本。为应对这些挑战，我们提出了 Fire-Flyer AI-HPC 架构，这是一种硬件软件协同设计框架及其最佳实践。对于 DL 训练，我们在 Fire-Flyer 2 上部署了 10,000 块 PCIe A100 GPU，实现的性能接近 DGX-A100，同时将成本降低了一半，能耗降低了 40%。我们专门设计 HFReduce 以加速 allreduce 通信，并实施多项措施确保计算存储集成网络保持拥塞-free。通过我们的软件堆栈，包括 HaiScale、3FS 和 HAI-Platform，我们实现了计算与通信的大规模重叠。我们的系统化经验为推动 AI-HPC 未来发展提供了宝贵见解。
引用
@article{arxiv.2408.14158,
title = {Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning},
author = {Wei An and Xiao Bi and Guanting Chen and Shanhuang Chen and Chengqi Deng and Honghui Ding and Kai Dong and Qiushi Du and Wenjun Gao and Kang Guan and Jianzhong Guo and Yongqiang Guo and Zhe Fu and Ying He and Panpan Huang and Jiashi Li and Wenfeng Liang and Xiaodong Liu and Xin Liu and Yiyuan Liu and Yuxuan Liu and Shanghao Lu and Xuan Lu and Xiaotao Nie and Tian Pei and Junjie Qiu and Hui Qu and Zehui Ren and Zhangli Sha and Xuecheng Su and Xiaowen Sun and Yixuan Tan and Minghui Tang and Shiyu Wang and Yaohui Wang and Yongji Wang and Ziwei Xie and Yiliang Xiong and Yanhong Xu and Shengfeng Ye and Shuiping Yu and Yukun Zha and Liyue Zhang and Haowei Zhang and Mingchuan Zhang and Wentao Zhang and Yichao Zhang and Chenggang Zhao and Yao Zhao and Shangyan Zhou and Shunfeng Zhou and Yuheng Zou},
journal= {arXiv preprint arXiv:2408.14158},
year = {2024}
}
备注
This is the preprint version of the paper accepted for presentation at the 2024 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'24). \c{opyright} 2024 IEEE. Personal use of this material is permitted. For other uses, permission from IEEE must be obtained. Please refer to IEEE Xplore for the final published version