In recent years, an active field of research has developed around automated machine learning (AutoML). Unfortunately, comparing different AutoML systems is hard and often done incorrectly. We introduce an open, ongoing, and extensible benchmark framework which follows best practices and avoids common mistakes. The framework is open-source, uses public datasets and has a website with up-to-date results. We use the framework to conduct a thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.
@article{arxiv.1907.00909,
title = {An Open Source AutoML Benchmark},
author = {Pieter Gijsbers and Erin LeDell and Janek Thomas and Sébastien Poirier and Bernd Bischl and Joaquin Vanschoren},
journal= {arXiv preprint arXiv:1907.00909},
year = {2019}
}
Comments
Accepted paper at the AutoML Workshop at ICML 2019. Code: https://github.com/openml/automlbenchmark/ Accompanying website: https://openml.github.io/automlbenchmark/