Accelerating Compound LLM Training Workloads with Maestro
摘要
Compound LLM training workloads-such as knowledge distillation and multimodal LLM (MLLM) training-are gaining prominence. These typically comprise heterogeneous components differing in parameter scale, execution mode (forward-only or full forward-backward), and sequence length. Besides, component activation can be data-dependent: in MLLM training, modality-specific parts activate only when inputs contain corresponding modalities, causing dynamic computational paths and irregular runtime workloads. Conventional frameworks, designed for monolithic models, cannot handle the dual heterogeneity-static (across components) and dynamic (runtime). By enforcing one-size-fits-all training configurations across components and ignoring input-induced variations, they suffer suboptimal throughput and poor GPU utilization. In this paper, we introduce Maestro, a section-centric training framework that addresses both challenges. Maestro first restructures the workload into a coarse-grained section graph. Each section independently configures its parallelism strategy, micro-batch size, and data-parallel degree-enabling fine-grained, component-aware resource allocation to tackle static heterogeneity. To tackle runtime irregularity, Maestro introduces a wavefront scheduling algorithm that dynamically reorders input samples to orchestrate concurrent section execution while preserving cross-section data dependencies. This maximizes inter-section parallelism and minimizes stalls, boosting hardware utilization. Deployed in production for millions of GPU hours, Maestro reduces GPU consumption by ~40% on key workloads-including knowledge distillation and MLLM training-validating its real-world impact.
引用
@article{arxiv.2605.10501,
title = {Accelerating Compound LLM Training Workloads with Maestro},
author = {Xiulong Yuan and Hongqing Chen and Jiaxuan Peng and Fan Zhou and Zhixiang Ruan and Zekun Wang and Bo Zheng and Rui Men and Haiquan Wang and Zhipeng Zhang and Langshi Chen and Man Yuan and Jiaqi Gao and Zhengping Qian and Junyang Lin and Yong Li and Wei Lin and Junhua Wang and Jingren Zhou},
journal= {arXiv preprint arXiv:2605.10501},
year = {2026}
}