Formalizing the Problem of Side Effect Regularization
Abstract
AI objectives are often hard to specify properly. Some approaches tackle this problem by regularizing the AI's side effects: Agents must weigh off "how much of a mess they make" with an imperfectly specified proxy objective. We propose a formal criterion for side effect regularization via the assistance game framework. In these games, the agent solves a partially observable Markov decision process (POMDP) representing its uncertainty about the objective function it should optimize. We consider the setting where the true objective is revealed to the agent at a later time step. We show that this POMDP is solved by trading off the proxy reward with the agent's ability to achieve a range of future tasks. We empirically demonstrate the reasonableness of our problem formalization via ground-truth evaluation in two gridworld environments.
Keywords
Cite
@article{arxiv.2206.11812,
title = {Formalizing the Problem of Side Effect Regularization},
author = {Alexander Matt Turner and Aseem Saxena and Prasad Tadepalli},
journal= {arXiv preprint arXiv:2206.11812},
year = {2022}
}
Comments
14 pages, accepted to ML Safety Workshop at NeurIPS 2022. Alexander Turner and Aseem Saxena contributed equally