To encourage the development of methods with reproducible and robust training behavior, we propose a challenge paradigm where competitors are evaluated directly on the performance of their learning procedures rather than pre-trained agents. Since competition organizers re-train proposed methods in a controlled setting they can guarantee reproducibility, and -- by retraining submissions using a held-out test set -- help ensure generalization past the environments on which they were trained.
@article{arxiv.2005.06041,
title = {Guaranteeing Reproducibility in Deep Learning Competitions},
author = {Brandon Houghton and Stephanie Milani and Nicholay Topin and William Guss and Katja Hofmann and Diego Perez-Liebana and Manuela Veloso and Ruslan Salakhutdinov},
journal= {arXiv preprint arXiv:2005.06041},
year = {2020}
}
Comments
Accepted as a poster presentation to the 2019 NeruIPS Challenges in Machine Learning workshop (CiML)