Large-scale ML training jobs are frequently interrupted by hardware and software anomalies, failures, and management events. Existing solutions like checkpoint-restart or runtime reconfiguration suffer from long downtimes and degraded performance. We present TrainMover, a resilient LLM training runtime that leverages elastic and standby machines to handle interruptions with minimal downtime and zero memory overhead. To achieve these goals, TrainMover introduces three key techniques: two-phase, delta-based communication group setup; communication-free sandboxed warmup; and general standby design that enables failure recovery from any role. Our evaluation shows that TrainMover consistently achieves around 20 seconds of downtime when handling various interruptions at the 1024-GPU scale. TrainMover is projected to reduce wasted GPU hours by 55% compared to the best alternative, saving 1.4 million GPU-hours per week at the 64K-GPU scale.
@article{arxiv.2412.12636,
title = {TrainMover: An Interruption-Resilient Runtime for ML Training},
author = {ChonLam Lao and Jiaqi Gao and Jiamin Cao and Zhipeng Zhang and Pengcheng Zhang and Jiangfei Duan and Zhilong Zheng and Yu Guan and Yichi Xu and Yong Li and Zhengping Qian and Aditya Akella and Minlan Yu and Ennan Zhai and Dennis Cai and Jingren Zhou},
journal= {arXiv preprint arXiv:2412.12636},
year = {2026}
}