Recent advancements in reinforcement learning (RL) have shown promise for optimizing virtual machine scheduling (VMS) in small-scale clusters. The utilization of RL to large-scale cloud computing scenarios remains notably constrained. This paper introduces a scalable RL framework, called Cluster Value Decomposition Reinforcement Learning (CVD-RL), to surmount the scalability hurdles inherent in large-scale VMS. The CVD-RL framework innovatively combines a decomposition operator with a look-ahead operator to adeptly manage representation complexities, while complemented by a Top-k filter operator that refines exploration efficiency. Different from existing approaches limited to clusters of 10 or fewer physical machines (PMs), CVD-RL extends its applicability to environments encompassing up to 50 PMs. Furthermore, the CVD-RL framework demonstrates generalization capabilities that surpass contemporary SOTA methodologies across a variety of scenarios in empirical studies. This breakthrough not only showcases the framework's exceptional scalability and performance but also represents a significant leap in the application of RL for VMS within complex, large-scale cloud infrastructures. The code is available at https://anonymous.4open.science/r/marl4sche-D0FE.
@article{arxiv.2503.00537,
title = {Scalable Reinforcement Learning for Virtual Machine Scheduling},
author = {Junjie Sheng and Jiehao Wu and Haochuan Cui and Yiqiu Hu and Wenli Zhou and Lei Zhu and Qian Peng and Wenhao Li and Xiangfeng Wang},
journal= {arXiv preprint arXiv:2503.00537},
year = {2025}
}