English

StreamFlow: cross-breeding cloud with HPC

Distributed, Parallel, and Cluster Computing 2024-07-09 v3

Abstract

Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.

Keywords

Cite

@article{arxiv.2002.01558,
  title  = {StreamFlow: cross-breeding cloud with HPC},
  author = {Iacopo Colonnelli and Barbara Cantalupo and Ivan Merelli and Marco Aldinucci},
  journal= {arXiv preprint arXiv:2002.01558},
  year   = {2024}
}

Comments

30 pages - 2020 IEEE Transactions on Emerging Topics in Computing