Model-Free Risk-Sensitive Reinforcement Learning
Machine Learning
2021-11-05 v1
Abstract
We extend temporal-difference (TD) learning in order to obtain risk-sensitive, model-free reinforcement learning algorithms. This extension can be regarded as modification of the Rescorla-Wagner rule, where the (sigmoidal) stimulus is taken to be either the event of over- or underestimating the TD target. As a result, one obtains a stochastic approximation rule for estimating the free energy from i.i.d. samples generated by a Gaussian distribution with unknown mean and variance. Since the Gaussian free energy is known to be a certainty-equivalent sensitive to the mean and the variance, the learning rule has applications in risk-sensitive decision-making.
Cite
@article{arxiv.2111.02907,
title = {Model-Free Risk-Sensitive Reinforcement Learning},
author = {Grégoire Delétang and Jordi Grau-Moya and Markus Kunesch and Tim Genewein and Rob Brekelmans and Shane Legg and Pedro A. Ortega},
journal= {arXiv preprint arXiv:2111.02907},
year = {2021}
}
Comments
DeepMind Tech Report: 13 pages, 4 figures