English

End-to-End Predictions-Based Resource Management Framework for Supercomputer Jobs

Distributed, Parallel, and Cluster Computing 2020-08-20 v1

Abstract

Job submissions of parallel applications to production supercomputer systems will have to be carefully tuned in terms of the job submission parameters to obtain minimum response times. In this work, we have developed an end-to-end resource management framework that uses predictions of queue waiting and execution times to minimize response times of user jobs submitted to supercomputer systems. Our method for predicting queue waiting times adaptively chooses a prediction method based on the cluster structure of similar jobs. Our strategy for execution time predictions dynamically learns the impact of load on execution times and uses this to predict a set of execution time ranges for the target job. We have developed two resource management techniques that employ these predictions, one that selects the number of processors for execution and the other that also dynamically changes the job submission time. Using workload simulations of large supercomputer traces, we show large-scale improvements in predictions and reductions in response times over existing techniques and baseline strategies.

Keywords

Cite

@article{arxiv.2008.08292,
  title  = {End-to-End Predictions-Based Resource Management Framework for Supercomputer Jobs},
  author = {Swetha Hariharan and Prakash Murali and Abhishek Pasari and Sathish Vadhiyar},
  journal= {arXiv preprint arXiv:2008.08292},
  year   = {2020}
}
R2 v1 2026-06-23T17:57:22.842Z