D2 Actor Critic：扩散 Actor 融合分布式 Critic
机器学习
2026-05-25 v3
摘要
我们引入 D2AC，一种新的无模型强化学习 (RL) 算法，旨在有效地在线训练表达性扩散策略。其核心是一个避免典型策略梯度高方差以及通过时间反向传播复杂度的策略改进目标。这种稳定的学习过程严格受益于我们的第二个贡献：一个稳健的分布式 Critic，我们通过将分布式 RL 与裁剪双 Q 学习融合来设计。 resulting algorithm is highly effective, achieving state-of-the-art performance on a benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate a biologically motivated predator-prey task to examine the behavioral robustness and generalization capacity of our approach. Code: https://github.com/d2ac-actor-critic/d2ac-public
引用
@article{arxiv.2510.03508,
title = {D2 Actor Critic: Diffusion Actor Meets Distributional Critic},
author = {Lunjun Zhang and Shuo Han and Hanrui Lyu and Bradly C Stadie},
journal= {arXiv preprint arXiv:2510.03508},
year = {2026}
}
备注
Accepted to TMLR 2025