跨越鸿沟：在马尔可夫采样下实现Actor-Critic的全局收敛（最后迭代）
摘要
当前Actor-Critic（AC）算法的前沿理论分析在 addressing 实际 AC 实现的实践方面存在显著滞后。这一关键差距需要填补，以将分析与 AC 的实际实现相一致。为此，我们提出了 MMCLG 标准：\textbf{M}ulti-layer neural network parametrization for actor/critic，\textbf{M}arkovian sampling，\textbf{C}ontinuous state-action spaces，the performance of the \textbf{L}ast iterate，and \textbf{G}lobal optimality。这些方面在实际中具有重要意义，且在现有 AC 算法的理论分析中 largely 被忽视。在本文中，我们通过提供涵盖所有五个关键实践方面（覆盖 MMCLG 标准）的 AC 算法的全面理论分析来弥补这些差距。我们建立了 的全局收敛样本复杂度上界。通过创新性地运用 MDP 的弱梯度支配属性以及对 critic 估计误差的独特分析，实现了这一结果。
引用
@article{arxiv.2405.01843,
title = {Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization},
author = {Mudit Gaur and Amrit Singh Bedi and Di Wang and Vaneet Aggarwal},
journal= {arXiv preprint arXiv:2405.01843},
year = {2024}
}
备注
Accepted at ICML 2024. This is a revised version of arXiv:2306.10486, where we have gone from finite action space to continuous action space, from average iterate convergence to last iterate convergence and from $\epsilon^{-4}$ to $\epsilon^{-3}$ sample complexity. This version fixes the related work result of (Xu et al., 2020a), based on their result update on arXiv