Using Experience Classification for Training Non-Markovian Tasks
Abstract
Unlike the standard Reinforcement Learning (RL) model, many real-world tasks are non-Markovian, whose rewards are predicated on state history rather than solely on the current state. Solving a non-Markovian task, frequently applied in practical applications such as autonomous driving, financial trading, and medical diagnosis, can be quite challenging. We propose a novel RL approach to achieve non-Markovian rewards expressed in temporal logic LTL (Linear Temporal Logic over Finite Traces). To this end, an encoding of linear complexity from LTL into MDPs (Markov Decision Processes) is introduced to take advantage of advanced RL algorithms. Then, a prioritized experience replay technique based on the automata structure (semantics equivalent to LTL specification) is utilized to improve the training process. We empirically evaluate several benchmark problems augmented with non-Markovian tasks to demonstrate the feasibility and effectiveness of our approach.
Cite
@article{arxiv.2310.11678,
title = {Using Experience Classification for Training Non-Markovian Tasks},
author = {Ruixuan Miao and Xu Lu and Cong Tian and Bin Yu and Zhenhua Duan},
journal= {arXiv preprint arXiv:2310.11678},
year = {2023}
}