Near-optimal Optimistic Reinforcement Learning using Empirical Bernstein Inequalities
Abstract
We study model-based reinforcement learning in an unknown finite communicating Markov decision process. We propose a simple algorithm that leverages a variance based confidence interval. We show that the proposed algorithm, UCRL-V, achieves the optimal regret up to logarithmic factors, and so our work closes a gap with the lower bound without additional assumptions on the MDP. We perform experiments in a variety of environments that validates the theoretical bounds as well as prove UCRL-V to be better than the state-of-the-art algorithms.
Cite
@article{arxiv.1905.12425,
title = {Near-optimal Optimistic Reinforcement Learning using Empirical Bernstein Inequalities},
author = {Aristide Tossou and Debabrota Basu and Christos Dimitrakakis},
journal= {arXiv preprint arXiv:1905.12425},
year = {2019}
}
Comments
the algorithm has been simplified (no need to look at lower bound of the reward and transitions). Proof has been significantly clean-up. The previous "assumption" is clarified as a condition of the algorithm well-known as sub-modularity. The proof that the bounds satisfy the submodularity is clean-up