KL-learning: Online solution of Kullback-Leibler control problems
Optimization and Control
2012-02-17 v2 Artificial Intelligence
Abstract
We introduce a stochastic approximation method for the solution of an ergodic Kullback-Leibler control problem. A Kullback-Leibler control problem is a Markov decision process on a finite state space in which the control cost is proportional to a Kullback-Leibler divergence of the controlled transition probabilities with respect to the uncontrolled transition probabilities. The algorithm discussed in this work allows for a sound theoretical analysis using the ODE method. In a numerical experiment the algorithm is shown to be comparable to the power method and the related Z-learning algorithm in terms of convergence speed. It may be used as the basis of a reinforcement learning style algorithm for Markov decision problems.
Cite
@article{arxiv.1112.1996,
title = {KL-learning: Online solution of Kullback-Leibler control problems},
author = {Joris Bierkens and Bert Kappen},
journal= {arXiv preprint arXiv:1112.1996},
year = {2012}
}