Related papers: Global Optimization of Data Pipelines in Heterogen…
Many scientific workflows can be represented by a Directed Acyclic Graph (DAG) where each node represents a task, and there will be a directed edge between two tasks if and only if there is a dependency relationship between the two i.e. the…
Many real-world scientific workflows can be represented by a Directed Acyclic Graph (DAG), where each node represents a task and a directed edge signifies a dependency between two tasks. Due to the increasing computational resource…
The analysis of massive scientific data often happens in the form of workflows with interdependent tasks. When such a scientific workflow needs to be scheduled on a parallel or distributed system, one usually represents the workflow as a…
Large data and computing centers consume a significant share of the world's energy consumption. A prominent subset of the workloads in such centers are workflows with interdependent tasks, usually represented as directed acyclic graphs…
When orchestrating Web service workflows, the geographical placement of the orchestration engine(s) can greatly affect workflow performance. Data may have to be transferred across long geographical distances, which in turn increases…
Cloud infrastructure supports the efficient operation of data pipelines regarding requirements like cost, speed, and resource utilization. We present an integrated view of optimization opportunities for cloud-based data pipelines by…
NextG (5G and beyond) networks, through the increasing integration of cloud/edge computing technologies, are becoming highly distributed compute platforms ideally suited to host emerging resource-intensive and latency-sensitive applications…
This work proposes a competitive scheduling approach, designed to scale to large heterogeneous multicore systems. This scheduler overcomes the challenges of (1) the high computation overhead of near-optimal schedulers, and (2) the error…
Scientific workflows are often represented as directed acyclic graphs (DAGs), where vertices correspond to tasks and edges represent the dependencies between them. Since these graphs are often large in both the number of tasks and their…
Many scientific workflows can be modeled as a Directed Acyclic Graph (henceforth mentioned as DAG) where the nodes represent individual tasks and the directed edges represent data and control flow dependency between two tasks. Due to large…
Hard real-time systems like image processing, autonomous driving, etc. require an increasing need of computational power that classical multi-core platforms can not provide, to fulfill with their timing constraints. Heterogeneous…
Python-written data analytics applications can be modeled as and compiled into a directed acyclic graph (DAG) based workflow, where the nodes are fine-grained tasks and the edges are task dependencies. Such analytics workflow jobs are…
Applications in data-parallel computing typically consist of multiple stages. In each stage, a set of intermediate parallel data flows (Coflow) is produced and transferred between servers to enable starting of next stage. While there has…
The proliferation of camera-enabled devices and large video repositories has led to a diverse set of video analytics applications. These applications rely on video pipelines, represented as DAGs of operations, to transform videos, process…
The convergence of IoT, Edge, Cloud, and HPC technologies creates a compute continuum that merges cloud scalability and flexibility with HPC's computational power and specialized optimizations. However, integrating cloud and HPC resources…
Scheduling job flows efficiently and rapidly on distributed computing clusters is one of huge challenges for daily operation of data centers. In a practical scenario, a single job consists of numerous stages with complex dependency relation…
Real-time AI services increasingly operate across the device-edge-cloud continuum, where autonomous AI agents generate latency-sensitive workloads, orchestrate multi-stage processing pipelines, and compete for shared resources under policy…
AI agents execute tens to hundreds of chained LLM calls per task, yet GPU schedulers treat each call as independent, discarding gigabytes of intermediate state between steps and inflating end-to-end latency by 3-8x. We argue that this…
This report is part of the DataflowOpt project on optimization of modern dataflows and aims to introduce a data quality-aware cost model that covers the following aspects in combination: (1) heterogeneity in compute nodes, (2)…
Recent breakthroughs in generative artificial intelligence have triggered a surge in demand for machine learning training, which poses significant cost burdens and environmental challenges due to its substantial energy consumption.…