English

Depth and nonlinearity induce implicit exploration for RL

Machine Learning 2018-05-31 v1 Artificial Intelligence Machine Learning

Abstract

The question of how to explore, i.e., take actions with uncertain outcomes to learn about possible future rewards, is a key question in reinforcement learning (RL). Here, we show a surprising result: We show that Q-learning with nonlinear Q-function and no explicit exploration (i.e., a purely greedy policy) can learn several standard benchmark tasks, including mountain car, equally well as, or better than, the most commonly-used ϵ\epsilon-greedy exploration. We carefully examine this result and show that both the depth of the Q-network and the type of nonlinearity are important to induce such deterministic exploration.

Keywords

Cite

@article{arxiv.1805.11711,
  title  = {Depth and nonlinearity induce implicit exploration for RL},
  author = {Justas Dauparas and Ryota Tomioka and Katja Hofmann},
  journal= {arXiv preprint arXiv:1805.11711},
  year   = {2018}
}
R2 v1 2026-06-23T02:12:38.667Z