Standing Together for Reproducibility in Large-Scale Computing: Report on reproducibility@XSEDE
Abstract
This is the final report on reproducibility@xsede, a one-day workshop held in conjunction with XSEDE14, the annual conference of the Extreme Science and Engineering Discovery Environment (XSEDE). The workshop's discussion-oriented agenda focused on reproducibility in large-scale computational research. Two important themes capture the spirit of the workshop submissions and discussions: (1) organizational stakeholders, especially supercomputer centers, are in a unique position to promote, enable, and support reproducible research; and (2) individual researchers should conduct each experiment as though someone will replicate that experiment. Participants documented numerous issues, questions, technologies, practices, and potentially promising initiatives emerging from the discussion, but also highlighted four areas of particular interest to XSEDE: (1) documentation and training that promotes reproducible research; (2) system-level tools that provide build- and run-time information at the level of the individual job; (3) the need to model best practices in research collaborations involving XSEDE staff; and (4) continued work on gateways and related technologies. In addition, an intriguing question emerged from the day's interactions: would there be value in establishing an annual award for excellence in reproducible research?
Cite
@article{arxiv.1412.5557,
title = {Standing Together for Reproducibility in Large-Scale Computing: Report on reproducibility@XSEDE},
author = {Doug James and Nancy Wilkins-Diehr and Victoria Stodden and Dirk Colbry and Carlos Rosales and Mark Fahey and Justin Shi and Rafael F. Silva and Kyo Lee and Ralph Roskies and Laurence Loewe and Susan Lindsey and Rob Kooper and Lorena Barba and David Bailey and Jonathan Borwein and Oscar Corcho and Ewa Deelman and Michael Dietze and Benjamin Gilbert and Jan Harkes and Seth Keele and Praveen Kumar and Jong Lee and Erika Linke and Richard Marciano and Luigi Marini and Chris Mattman and Dave Mattson and Kenton McHenry and Robert McLay and Sheila Miguez and Barbara Minsker and Maria Perez-Hernandez and Dan Ryan and Mats Rynge and Idafen Santana-Perez and Mahadev Satyanarayanan and Gloriana St. Clair and Keith Webster and Elvind Hovig and Daniel S. Katz and Sophie Kay and Geir Sandve and David Skinner and Gabrielle Allen and John Cazes and Kym Won Cho and Jim Fonseca and Lorraine Hwang and Lars Koesterke and Pragnesh Patel and Line Pouchard and Ed Seidel and Isuru Suriarachchi},
journal= {arXiv preprint arXiv:1412.5557},
year = {2015}
}