Scientific research increasingly relies on distributed computational resources, storage systems, networks, and instruments, ranging from HPC and cloud systems to edge devices. Event-driven architecture (EDA) benefits applications targeting distributed research infrastructures by enabling the organization, communication, processing, reliability, and security of events generated from many sources. To support the development of scientific EDA, we introduce Octopus, a hybrid, cloud-to-edge event fabric designed to link many local event producers and consumers with cloud-hosted brokers. Octopus can be scaled to meet demand, permits the deployment of highly available Triggers for automatic event processing, and enforces fine-grained access control. We identify requirements in self-driving laboratories, scientific data automation, online task scheduling, epidemic modeling, and dynamic workflow management use cases, and present results demonstrating Octopus' ability to meet those requirements. Octopus supports producing and consuming events at a rate of over 4.2 M and 9.6 M events per second, respectively, from distributed clients.
@article{arxiv.2407.11432,
title = {Octopus: Experiences with a Hybrid Event-Driven Architecture for Distributed Scientific Computing},
author = {Haochen Pan and Ryan Chard and Sicheng Zhou and Alok Kamatar and Rafael Vescovi and Valérie Hayot-Sasson and André Bauer and Maxime Gonthier and Kyle Chard and Ian Foster},
journal= {arXiv preprint arXiv:2407.11432},
year = {2024}
}
Comments
12 pages and 8 figures. Camera-ready version for FTXS'24 (https://sites.google.com/view/ftxs2024)