Resilient AI Supercomputer Networking using MRC and SRv6
Abstract
Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, allowing training clusters well over 100K GPUs to be built as two-tier topologies while increasing physical redundancy, and (3) the use of static source-routing using SRv6 to allow MRC the freedom to bypass failures by itself. We describe our experiences running MRC and static SRv6 routing in production in OpenAI and Microsoft's largest training clusters, where it has been used to train the latest frontier models. We demonstrate how MRC allows AI training jobs to ride out many network failures that previously would have interrupted training.
Cite
@article{arxiv.2605.04333,
title = {Resilient AI Supercomputer Networking using MRC and SRv6},
author = {Joao Araujo and Alex Chow and Mark Handley and Ryder Lewis and Christoph Paasch and Jitendra Padhye and Michael Papamichael and Greg Steinbrecher and Amin Tootoonchian and Lihua Yuan and S. Anantharamu and Abhishek Dosi and Mohit Garg and Mahdieh Ghazi and Torsten Hoefler and Deepal Jayasinghe and Jithin Jose and Abdul Kabbani and Guohan Lu and Yang Wang and K. Doddapaneni and Murali Garimella and Vipin Jain and Yanfang Le and H. Nagulapalli and S. Narayanan and Rong Pan and Rathina Sabesan and Raghava Sivaramu and Rip Sohan and Eric Davis and Dragos Dumitrescu and Mohan Kalkunte and Bhaswar Mitra and Guglielmo Morandin and Adrian Popa and Costin Raiciu and Eric Spada and John Spillane and Niranjan Vaidya and Aviv Barnea and Idan Burstein and Elazar Cohen and Yamin Friedman and Noam Katz and Masoud Moshref and Yuval Shpigelman and Shahaf Shuler and Shy Shyman and Sayantan Sur},
journal= {arXiv preprint arXiv:2605.04333},
year = {2026}
}
Comments
18 pages, 22 figures