Distributed applications, such as database queries and distributed training, consist of both compute and network tasks. DAG-based abstraction primarily targets compute tasks and has no explicit network-level scheduling. In contrast, Coflow abstraction collectively schedules network flows among compute tasks but lacks the end-to-end view of the application DAG. Because of the dependencies and interactions between these two types of tasks, it is sub-optimal to only consider one of them. We argue that co-scheduling of both compute and network tasks can help applications towards the globally optimal end-to-end performance. However, none of the existing abstractions can provide fine-grained information for co-scheduling. We propose MXDAG, an abstraction to treat both compute and network tasks explicitly. It can capture the dependencies and interactions of both compute and network tasks leading to improved application performance.
@article{arxiv.2107.07442,
title = {MXDAG: A Hybrid Abstraction for Cluster Applications},
author = {Weitao Wang and Sushovan Das and Xinyu Crystal Wu and Zhuang Wang and Ang Chen and T. S. Eugene Ng},
journal= {arXiv preprint arXiv:2107.07442},
year = {2021}
}