Fault-tolerance has always been an important topic when it comes to running massively parallel programs at scale. Statistically, hardware and software failures are expected to occur more often on systems gathering millions of computing units. Moreover, the larger jobs are, the more computing hours would be wasted by a crash. In this paper, we describe the work done in our MPI runtime to enable both transparent and application-level checkpointing mechanisms. Unlike the MPI 4.0 User-Level Failure Mitigation (ULFM) interface, our work targets solely Checkpoint/Restart and ignores other features such as resiliency. We show how existing checkpointing methods can be practically applied to a thread-based MPI implementation given sufficient runtime collaboration. The two main contributions are the preservation of high-speed network performance during transparent C/R and the over-subscription of checkpoint data replication thanks to a dedicated user-level scheduler support. These techniques are measured on MPI benchmarks such as IMB, Lulesh and Heatdis, and associated overhead and trade-offs are discussed.
@article{arxiv.1906.05020,
title = {Checkpoint/restart approaches for a thread-based MPI runtime},
author = {Julien Adam and Maxime Kermarquer and Jean-Baptiste Besnard and Leonardo Bautista-Gomez and Marc Perache and Patrick Carribault and Julien Jaeger and Allen D. Malony and Sameer Shende},
journal= {arXiv preprint arXiv:1906.05020},
year = {2019}
}
Comments
This research has been partially sponsored by the European Union s Horizon 2020 Programme under the LEGaTO Project (www.legato-project.eu), grant agreement 780681 and the Mont-Blanc 2020 project, grant agreement no. 779877