Many scientific high-throughput applications can benefit from the elastic nature of Cloud resources, especially when there is a need to reduce time to completion. Cost considerations are usually a major issue in such endeavors, with networking often a major component; for data-intensive applications, egress networking costs can exceed the compute costs. Dedicated network links provide a way to lower the networking costs, but they do add complexity. In this paper we provide a description of a 100 fp32 PFLOPS Cloud burst in support of IceCube production compute, that used Internet2 Cloud Connect service to provision several logically-dedicated network links from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and Google Cloud Platform, that in aggregate enabled approximately 100 Gbps egress capability to on-prem storage. It provides technical details about the provisioning process, the benefits and limitations of such a setup and an analysis of the costs incurred.
@article{arxiv.2104.06913,
title = {Managing Cloud networking costs for data-intensive applications by provisioning dedicated network links},
author = {Igor Sfiligoi and Michael Hare and David Schultz and Frank Würthwein and Benedikt Riedel and Tom Hutton and Steve Barnet and Vladimir Brik},
journal= {arXiv preprint arXiv:2104.06913},
year = {2021}
}
Comments
8 pages, 7 figures, 4 tables, to be published in proceedings of PEARC21