Context: Systematic reviews (SRs) summarize state-of-the-art evidence in science, including software engineering (SE). Objective: Our objective is to evaluate how SRs report research artifacts and to provide a comprehensive list of these artifacts. Method: We examined 537 secondary studies published between 2013 and 2023 to analyze the availability and reporting of research artifacts. Results: Our findings indicate that only 31.5% of the reviewed studies include research artifacts. Encouragingly, the situation is gradually improving, as our regression analysis shows a significant increase in the availability of research artifacts over time. However, in 2023, just 62.0% of secondary studies provide a research artifact while an even lower percentage, 30.4% use a permanent repository with a digital object identifier (DOI) for storage. Conclusion: To enhance transparency and reproducibility in SE research, we advocate for the mandatory publication of research artifacts in secondary studies.
@article{arxiv.2504.12646,
title = {Research Artifacts in Secondary Studies: A Systematic Mapping in Software Engineering},
author = {Aleksi Huotala and Miikka Kuutila and Mika Mäntylä},
journal= {arXiv preprint arXiv:2504.12646},
year = {2026}
}
Comments
Published in Information and Software Technology (Volume 187, November 2025, 107830)