On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measures
Abstract
Risk-sensitive reinforcement learning (RL) has become a popular tool for controlling the risk of uncertain outcomes and ensuring reliable performance in highly stochastic sequential decision-making problems. While it has been shown that policy gradient methods can find globally optimal policies in the risk-neutral setting, it remains unclear if the risk-averse variants enjoy the same global convergence guarantees. In this paper, we consider a class of dynamic time-consistent risk measures, named Expected Conditional Risk Measures (ECRMs), and derive natural policy gradient (NPG) updates for ECRMs-based RL problems. We provide global optimality and iteration complexity of the proposed risk-averse NPG algorithm with softmax parameterization and entropy regularization under both exact and inexact policy evaluation. Furthermore, we test our risk-averse NPG algorithm on a stochastic Cliffwalk environment to demonstrate the efficacy of our method.
Keywords
Cite
@article{arxiv.2301.10932,
title = {On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measures},
author = {Xian Yu and Lei Ying},
journal= {arXiv preprint arXiv:2301.10932},
year = {2026}
}