Related papers: Terminologies for Reproducible Research
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which…
Over the past decade alongside increased focus on computational reproducibility significant efforts have been made to define reproducibility. However, these definitions provide a textual description rather than a framework. The community…
Scientific publications form the cornerstone of innovation and have maintained a stable growth trend over the years. However, in recent years, there has been a significant surge in retractions, driven largely by the proliferation of…
Two goals - improving replicability and accountability of Machine Learning research respectively, have accrued much attention from the AI ethics and the Machine Learning community. Despite sharing the measures of improving transparency, the…
Scientific knowledge is constantly subject to a variety of changes due to new discoveries, alternative interpretations, and fresh perspectives. Understanding uncertainties associated with various stages of scientific inquiries is an…
The crisis in the reproducibility of experiments invites a re-evaluation of methods of inquiry and validation procedures. The text challenges current assumptions of knowledge acquisition and introduces G-complexity for defining decidable…
In high-stakes information domains such as healthcare, where large language models (LLMs) can produce hallucinations or misinformation, retrieval-augmented generation (RAG) has been proposed as a mitigation strategy, grounding model outputs…
Natural language counterfactual generation aims to minimally modify a given text such that the modified text will be classified into a different class. The generated counterfactuals provide insight into the reasoning behind a model's…
Difficulties in replication and reproducibility of empirical evidences in machine learning research have become a prominent topic in recent years. Ensuring that machine learning research results are sound and reliable requires…
Academic disciplines, often divided into hard and soft sciences, may be understood as "donor disciplines" if they produce more concepts than they borrow from other disciplines, or "borrower disciplines" if they import more than they…
Collaborations are an integral part of scientific research and publishing. In the past, access to large-scale corpora has limited the ways in which questions about collaborations could be investigated. However, with improvements in…
As software has become an integral part of scientific workflows, reproducible research practices must take it into account. In what way? Archiving source code is a necessary but insufficient condition. The ability to redeploy software…
We investigate replicable learning algorithms. Ideally, we would like to design algorithms that output the same canonical model over multiple runs, even when different runs observe a different set of samples from the unknown data…
Scientists in some fields are concerned that many, or even most, published results are false. A high rate of false positives might arise accidentally, from shoddy research practices. Or it might be the inevitable result of institutional…
Competitive debaters often find themselves facing a challenging task -- how to debate a topic they know very little about, with only minutes to prepare, and without access to books or the Internet? What they often do is rely on "first…
The tension between qualitative theorizing and quantitative methods is pervasive in the social sciences, and poses a constant challenge to empirical research. But in science studies as an interdisciplinary specialty, there are additional…
We analyze the publication records of individual scientists, aiming to quantify the topic switching dynamics of scientists and its influence. For each scientist, the relations among her publications are characterized via shared references.…
Collaboration between health science and visual analytics research is often hindered by different, sometimes incompatible approaches to research design. Health science often follows hypothesis-driven protocols, registered in advance, and…
As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other…
Reproducibility is a fundamental requirement of the scientific process since it enables outcomes to be replicated and verified. Computational scientific experiments can benefit from improved reproducibility for many reasons, including…