Scaling up deep learning models has been proven effective to improve intelligence of machine learning (ML) models, especially for industry recommendation models and large language models. The co-design of large distributed ML systems and algorithms (to maximize training performance) plays a pivotal role for its success. As it scales, the number of co-design hyper-parameters grows rapidly which brings challenges to feasibly find the optimal setup for system performance maximization. In this paper, we propose CubicML which uses ML to automatically optimize training performance of large distributed ML systems. In CubicML, we use an ML model as a proxy to predict the training performance for search efficiency and performance modeling flexibility. We proved that CubicML can effectively optimize training speed of in-house ads recommendation models with 73 billion parameters and large language models up to 405 billion parameters at Meta.
@article{arxiv.2409.04585,
title = {CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performance},
author = {Wei Wen and Quanyu Zhu and Weiwei Chu and Wen-Yen Chen and Jiyan Yang},
journal= {arXiv preprint arXiv:2409.04585},
year = {2024}
}