Benchmarking in Optimization: Best Practice and Open Issues
Abstract
This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.
Cite
@article{arxiv.2007.03488,
title = {Benchmarking in Optimization: Best Practice and Open Issues},
author = {Thomas Bartz-Beielstein and Carola Doerr and Daan van den Berg and Jakob Bossek and Sowmya Chandrasekaran and Tome Eftimov and Andreas Fischbach and Pascal Kerschke and William La Cava and Manuel Lopez-Ibanez and Katherine M. Malan and Jason H. Moore and Boris Naujoks and Patryk Orzechowski and Vanessa Volz and Markus Wagner and Thomas Weise},
journal= {arXiv preprint arXiv:2007.03488},
year = {2020}
}
Comments
Version 2