A key challenge on the path to developing agents that learn complex human-like behavior is the need to quickly and accurately quantify human-likeness. While human assessments of such behavior can be highly accurate, speed and scalability are limited. We address these limitations through a novel automated Navigation Turing Test (ANTT) that learns to predict human judgments of human-likeness. We demonstrate the effectiveness of our automated NTT on a navigation task in a complex 3D environment. We investigate six classification models to shed light on the types of architectures best suited to this task, and validate them against data collected through a human NTT. Our best models achieve high accuracy when distinguishing true human and agent behavior. At the same time, we show that predicting finer-grained human assessment of agents' progress towards human-like behavior remains unsolved. Our work takes an important step towards agents that more effectively learn complex human-like behavior.
@article{arxiv.2105.09637,
title = {Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation},
author = {Sam Devlin and Raluca Georgescu and Ida Momennejad and Jaroslaw Rzepecki and Evelyn Zuniga and Gavin Costello and Guy Leroy and Ali Shaw and Katja Hofmann},
journal= {arXiv preprint arXiv:2105.09637},
year = {2021}
}
Comments
All data collected throughout this study, plus the code to reproduce our analysis and ANTT are available at https://github.com/microsoft/NTT