This paper tries to reduce the effort of learning, deploying, and integrating several frameworks for the development of e-Science applications that combine simulations with High-Performance Data Analytics (HPDA). We propose a way to extend task-based management systems to support continuous input and output data to enable the combination of task-based workflows and dataflows (Hybrid Workflows from now on) using a single programming model. Hence, developers can build complex Data Science workflows with different approaches depending on the requirements. To illustrate the capabilities of Hybrid Workflows, we have built a Distributed Stream Library and a fully functional prototype extending COMPSs, a mature, general-purpose, task-based, parallel programming model. The library can be easily integrated with existing task-based frameworks to provide support for dataflows. Also, it provides a homogeneous, generic, and simple representation of object and file streams in both Java and Python; enabling complex workflows to handle any data type without dealing directly with the streaming back-end.
@article{arxiv.2007.04939,
title = {A Programming Model for Hybrid Workflows: combining Task-based Workflows and Dataflows all-in-one},
author = {Cristian Ramon-Cortes and Francesc Lordan and Jorge Ejarque and Rosa M. Badia},
journal= {arXiv preprint arXiv:2007.04939},
year = {2020}
}
Comments
Accepted in Future Generation Computer Systems (FGCS). Licensed under CC-BY-NC-ND