Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss

Ukjo Hwang; Songnam Hong

Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss

Machine Learning 2025-04-15 v1

Authors: Ukjo Hwang , Songnam Hong

Abstract

Overestimation is a fundamental characteristic of model-free reinforcement learning (MF-RL), arising from the principles of temporal difference learning and the approximation of the Q-function. To address this challenge, we propose a novel moderate target in the Q-function update, formulated as a convex optimization of an overestimated Q-function and its lower bound. Our primary contribution lies in the efficient estimation of this lower bound through the lower expectile of the Q-value distribution conditioned on a state. Notably, our moderate target integrates seamlessly into state-of-the-art (SOTA) MF-RL algorithms, including Deep Deterministic Policy Gradient (DDPG) and Soft Actor Critic (SAC). Experimental results validate the effectiveness of our moderate target in mitigating overestimation bias in DDPG, SAC, and distributional RL algorithms.

Keywords

reinforcement learning multi-agent reinforcement learning

Cite

@article{arxiv.2504.09929,
  title  = {Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss},
  author = {Ukjo Hwang and Songnam Hong},
  journal= {arXiv preprint arXiv:2504.09929},
  year   = {2025}
}

Related papers

View all related →