Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark studies, this Perspective sheds light on a wish list for a computational platform to orchestrate benchmark studies. We discuss various ideas for organizing reproducible software environments, formally defining benchmarks, orchestrating standardized workflows, and how they interface with computing infrastructure.
@article{arxiv.2409.15472,
title = {Building a continuous benchmarking ecosystem in bioinformatics},
author = {Izaskun Mallona and Charlotte Soneson and Ben Carrillo and Almut Luetge and Daniel Incicau and Reto Gerber and Anthony Sonrel and Mark D. Robinson},
journal= {arXiv preprint arXiv:2409.15472},
year = {2025}
}