Task-Relevant Adversarial Imitation Learning
Abstract
We show that a critical vulnerability in adversarial imitation is the tendency of discriminator networks to learn spurious associations between visual features and expert labels. When the discriminator focuses on task-irrelevant features, it does not provide an informative reward signal, leading to poor task performance. We analyze this problem in detail and propose a solution that outperforms standard Generative Adversarial Imitation Learning (GAIL). Our proposed method, Task-Relevant Adversarial Imitation Learning (TRAIL), uses constrained discriminator optimization to learn informative rewards. In comprehensive experiments, we show that TRAIL can solve challenging robotic manipulation tasks from pixels by imitating human operators without access to any task rewards, and clearly outperforms comparable baseline imitation agents, including those trained via behaviour cloning and conventional GAIL.
Cite
@article{arxiv.1910.01077,
title = {Task-Relevant Adversarial Imitation Learning},
author = {Konrad Zolna and Scott Reed and Alexander Novikov and Sergio Gomez Colmenarejo and David Budden and Serkan Cabi and Misha Denil and Nando de Freitas and Ziyu Wang},
journal= {arXiv preprint arXiv:1910.01077},
year = {2020}
}
Comments
Accepted to CoRL 2020 (see presentation here: https://youtu.be/ZgQvFGuEgFU )