English

Beyond Pessimism: Offline Learning in KL-regularized Games

Computer Science and Game Theory 2026-05-11 v2 Machine Learning

Abstract

We study offline learning in KL-regularized two-player zero-sum games, where policies are optimized with respect to a fixed reference policy through KL regularization. Prior work relies on pessimistic value estimation to handle distribution shift, yielding only O~(1/n)\widetilde{\mathcal{O}}(1/\sqrt n) statistical rates. We develop a new pessimism-free algorithm and analytical framework for KL-regularized games, built on the smoothness of KL-regularized best responses and a stability property of the Nash equilibrium induced by skew symmetry. This yields, to our knowledge, the first pessimism-free offline learning guarantee for KL-regularized games, with a fast O~(1/n)\widetilde{\mathcal{O}}(1/n) sample complexity bound. We further propose an efficient self-play policy optimization algorithm that replaces exact equilibrium computation with iterative KL-regularized policy updates, and prove that its last iterate preserves the same pessimism-free statistical guarantee up to a controlled optimization error.

Keywords

Cite

@article{arxiv.2604.06738,
  title  = {Beyond Pessimism: Offline Learning in KL-regularized Games},
  author = {Yuheng Zhang and Claire Chen and Nan Jiang},
  journal= {arXiv preprint arXiv:2604.06738},
  year   = {2026}
}
R2 v1 2026-07-01T11:58:44.828Z