On learning history based policies for controlling Markov decision processes
Abstract
Reinforcementlearning(RL)folkloresuggeststhathistory-basedfunctionapproximationmethods,suchas recurrent neural nets or history-based state abstraction, perform better than their memory-less counterparts, due to the fact that function approximation in Markov decision processes (MDP) can be viewed as inducing a Partially observable MDP. However, there has been little formal analysis of such history-based algorithms, as most existing frameworks focus exclusively on memory-less features. In this paper, we introduce a theoretical framework for studying the behaviour of RL algorithms that learn to control an MDP using history-based feature abstraction mappings. Furthermore, we use this framework to design a practical RL algorithm and we numerically evaluate its effectiveness on a set of continuous control tasks.
Cite
@article{arxiv.2211.03011,
title = {On learning history based policies for controlling Markov decision processes},
author = {Gandharv Patil and Aditya Mahajan and Doina Precup},
journal= {arXiv preprint arXiv:2211.03011},
year = {2022}
}