Coarse-Grained Smoothness for RL in Metric Spaces
Machine Learning
2021-10-26 v1
Abstract
Principled decision-making in continuous state--action spaces is impossible without some assumptions. A common approach is to assume Lipschitz continuity of the Q-function. We show that, unfortunately, this property fails to hold in many typical domains. We propose a new coarse-grained smoothness definition that generalizes the notion of Lipschitz continuity, is more widely applicable, and allows us to compute significantly tighter bounds on Q-functions, leading to improved learning. We provide a theoretical analysis of our new smoothness definition, and discuss its implications and impact on control and exploration in continuous domains.
Keywords
Cite
@article{arxiv.2110.12276,
title = {Coarse-Grained Smoothness for RL in Metric Spaces},
author = {Omer Gottesman and Kavosh Asadi and Cameron Allen and Sam Lobel and George Konidaris and Michael Littman},
journal= {arXiv preprint arXiv:2110.12276},
year = {2021}
}