Related papers: Standing Together for Reproducibility in Large-Sca…
GeoAI has emerged as an exciting interdisciplinary research area that combines spatial theories and data with cutting-edge AI models to address geospatial problems in a novel, data-driven manner. While GeoAI research has flourished in the…
Extreme edge computing (EEC) refers to the endmost part of edge computing wherein computational tasks and edge services are deployed only on extreme edge devices (EEDs). EEDs are consumer or user-owned devices that offer computational…
The TELOS Collaboration is committed to producing and analysing lattice data reproducibly, and sharing its research openly. In this document, we set out the ways that we make this happen, where there is scope for improvement, and how we…
Artificial Intelligence (AI) development is inherently iterative and experimental. Over the course of normal development, especially with the advent of automated AI, hundreds or thousands of experiments are generated and are often lost or…
Sharing research artifacts is known to help people to build upon existing knowledge, adopt novel contributions in practice, and increase the chances of papers receiving attention. In Model-Driven Engineering (MDE), openly providing research…
There are several forms of irreducibility in computing systems, ranging from undecidability to intractability to nonlinearity. This paper is an exploration of the conceptual issues that have arisen in the course of investigating speed-up…
While experimental reproduction remains a pillar of the scientific method, we observe that the software best practices supporting the reproduction of machine learning ( ML ) research are often undervalued or overlooked, leading both to poor…
Why are some research studies easy to reproduce while others are difficult? Casting doubt on the accuracy of scientific work is not fruitful, especially when an individual researcher cannot reproduce the claims made in the paper. There…
This report documents the process that led to the NSF Workshop on "Sustainable Computing for Sustainability" held in April 2024 at NSF in Alexandria, VA, and reports on its findings. The workshop's primary goals were to (i) advance the…
The Massive Text Embedding Benchmark (MTEB) has become a standard evaluation platform for text embedding models. While previous work has established the core benchmark methodology, this paper focuses on the engineering aspects that ensure…
Modern technologies are enabling scientists to collect extraordinary amounts of complex and sophisticated data across a huge range of scales like never before. With this onslaught of data, we can allow the focal point to shift towards…
University research groups in Computational Science and Engineering (CSE) generally lack dedicated funding and personnel for Research Software Engineering (RSE), which, combined with the pressure to maximize the number of scientific…
Reproducibility is inseparable from transparency, as sharing data, code and computational environment is a pre-requisite for being able to retrace the steps of producing the research results. Others have made the case that this artifact…
The Competition on Software Verification (SV-COMP) is a large computational experiment benchmarking many different software verification tools on a vast collection of C and Java benchmarks. Such experimental research should be reproducible…
Machine learning is facing a 'reproducibility crisis' where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of reproducibility failures using a meta-analysis…
Like other engineering disciplines, software engineering should also have principles to guide the construction of sustainable computer applications. Tangible properties include a) unlimited scalability, b) maximal reproducibility, and c)…
Scientific data governance should prioritize maximizing the utility of data throughout the research lifecycle. Research software systems that enable analysis reproducibility inform data governance policies and assist administrators in…
Background: This invited paper is the result of an invitation to write a retrospective article on a "TSE most influential paper" as part of the journal's 50th anniversary. Objective: To reflect on the progress of software engineering…
In this paper we reproduce experimental results presented in our earlier work titled "Describing Subjective Experiment Consistency by $p$-Value P-P Plot" that was presented in the course of the 28th ACM International Conference on…
Computational reproducibility is fundamental to trustworthy science, yet remains difficult to achieve in practice across various research workflows, including Jupyter notebooks published alongside scholarly articles. Environment drift,…