Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning
Abstract
Reinforcement Learning (RL) has demonstrated strong potential for industrial process control, yet policies trained in simulation often suffer from a significant sim-to-real gap when deployed on physical hardware. This work systematically analyzes how core Markov Decision Process (MDP) design choices -- state composition, target inclusion, reward formulation, termination criteria, and environment dynamics models -- affect this transfer. Using a color mixing task, we evaluate different MDP configurations and mixing dynamics across simulation and real-world experiments. We validate our findings on physical hardware, demonstrating that physics-based dynamics models achieve up to 50% real-world success under strict precision constraints where simplified models fail entirely. Our results provide practical MDP design guidelines for deploying RL in industrial process control.
Keywords
Cite
@article{arxiv.2603.09427,
title = {Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning},
author = {Tatjana Krau and Jorge Mandlmaier and Tobias Damm and Frieder Heieck},
journal= {arXiv preprint arXiv:2603.09427},
year = {2026}
}
Comments
This work has been submitted to the IEEE for possible publication