Related papers: Reproducibility and Statistical Methodology
Reproducibility is widely acknowledged as a fundamental principle in scientific research. Currently, the scientific community grapples with numerous challenges associated with reproducibility, often referred to as the ''reproducibility…
The TELOS Collaboration is committed to producing and analysing lattice data reproducibly, and sharing its research openly. In this document, we set out the ways that we make this happen, where there is scope for improvement, and how we…
We propose a novel method for selective classification (SC), a problem which allows a classifier to abstain from predicting some instances, thus trading off accuracy against coverage (the fraction of instances predicted). In contrast to…
In this paper peer review reliability is investigated based on peer ratings of research teams at two Belgian universities. It is found that outcomes can be substantially influenced by the different ways in which experts attribute ratings.…
Personal connections between creators and evaluators of scientific works are ubiquitous, and the possibility of bias ever-present. Although connections have been shown to bias prospective judgments of (uncertain) future performance, it is…
In this paper, we replicated a Bayesian educational research project, which explores the association between broadband access and online course enrollment in the US. We summarized key findings from our replication and compared them with the…
The ubiquity of computation in modern scientific research inflicts new challenges for reproducibility. While most journals now require code and data be made available, the standards for organization, annotation, and validation remain lax,…
Small to medium-scale data science experiments often rely on research software developed ad-hoc by individual scientists or small teams. Often there is no time to make the research software fast, reusable, and open access. The consequence…
Reproducibility has been consistently identified as an important component of scientific research. Although there is a general consensus on the importance of reproducibility along with the other commonly used 'R' terminology (i.e.,…
Scientists in some fields are concerned that many, or even most, published results are false. A high rate of false positives might arise accidentally, from shoddy research practices. Or it might be the inevitable result of institutional…
Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor…
The accuracy of published medical research is critical both for scientists, physicians and patients who rely on these results. But the fundamental belief in the medical literature was called into serious question by a paper suggesting most…
How many times have you tried to re-implement a past CAV tool paper, and failed? Reliably reproducing published scientific discoveries has been acknowledged as a barrier to scientific progress for some time but there remains only a small…
Data makes science possible. Sharing data improves visibility, and makes the research process transparent. This increases trust in the work, and allows for independent reproduction of results. However, a large proportion of data from…
Scientific progress in NLP rests on the reproducibility of researchers' claims. The *CL conferences created the NLP Reproducibility Checklist in 2020 to be completed by authors at submission to remind them of key information to include. We…
One of the main goals of Open Science is to make research more reproducible. There is no consensus, however, on what exactly "reproducibility" is, as opposed for example to "replicability", and how it applies to different research fields.…
Evaluating and comparing the academic performance of a journal, a researcher or a single paper has long remained a critical, necessary but also controversial issue. Most of existing metrics invalidate comparison across different fields of…
A new standard is proposed for the evidential assessment of replication studies. The approach combines a specific reverse-Bayes technique with prior-predictive tail probabilities to define replication success. The method gives rise to a…
Offline evaluation plays a central role in benchmarking recommender systems when online testing is impractical or risky. However, it is susceptible to two key sources of bias: exposure bias, where users only interact with items they are…
Nowadays, protecting trust in social sciences also means engaging in open community dialogue, which helps to safeguard robustness and improve efficiency of research methods. The combination of open data, open review and open dialogue may…