Related papers: A Case for a "Refutations and Critiques" Track in …
Peer reviewing is a central process in modern research and essential for ensuring high quality and reliability of published work. At the same time, it is a time-consuming process and increasing interest in emerging fields often results in a…
Many research fields are currently reckoning with issues of poor levels of reproducibility. Some label it a "crisis", and research employing or building Machine Learning (ML) models is no exception. Issues including lack of transparency,…
Reproducibility in research remains hindered by complex systems involving data, models, tools, and algorithms. Studies highlight a reproducibility crisis due to a lack of standardized reporting, code and data sharing, and rigorous…
Large language models offer a tempting solution to address the peer review crisis. This position paper argues that today's AI systems should not be used to produce paper reviews. We ground this position in an empirical comparison of human-…
In this paper peer review reliability is investigated based on peer ratings of research teams at two Belgian universities. It is found that outcomes can be substantially influenced by the different ways in which experts attribute ratings.…
Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale. Exponential growth in manuscript submissions to premier ML venues such as NeurIPS, ICML, and ICLR is outpacing the finite…
Over the past decade, a crisis of confidence in scientific literature has gained attention, particularly in the West. In response, we have seen changes in policy and practice amongst individual researchers and institutions. Greater…
Peer review is the main mechanism by which the software engineering community assesses the quality of scientific results. However, the rapid growth of paper submissions in software engineering venues has outpaced the availability of…
Over the past few years, deep learning methods have been applied for a wide range of Software Engineering (SE) tasks, including in particular for the important task of automatically predicting and localizing faults in software. With the…
Peer review is essential for scientific progress but faces growing challenges due to increasing submission volumes and reviewer fatigue. Existing automated review approaches struggle with factual accuracy, rating consistency, and analytical…
Scientific publications form the cornerstone of innovation and have maintained a stable growth trend over the years. However, in recent years, there has been a significant surge in retractions, driven largely by the proliferation of…
There has been increasing concern within the machine learning community that we are in a reproducibility crisis. As many have begun to work on this problem, all work we are aware of treat the issue of reproducibility as an intrinsic binary…
This essay delves into the ethical dilemmas encountered within the academic peer review process and investigates the prevailing deficiencies in this system. It highlights how established scholars often adhere to mainstream theories not out…
Peer review is a critical component of scientific progress in the fields like AI, but the rapid increase in submission volume has strained the reviewing system, which inevitably leads to reviewer shortages and declines review quality.…
Context: Empirical Software Engineering (ESE) drives innovation in SE through qualitative and quantitative studies. However, concerns about the correct application of empirical methodologies have existed since the 2006 Dagstuhl seminar on…
We propose an academic publishing system where research papers are stored in a network of data centres owned by university libraries and research institutions, and are interfaced with the academic community through a website. In our system,…
Explainably estimating confidence in published scholarly work offers opportunity for faster and more robust scientific progress. We develop a synthetic prediction market to assess the credibility of published claims in the social and…
Academic data sharing is a way for researchers to collaborate and thereby meet the needs of an increasingly complex research landscape. It enables researchers to verify results and to pursuit new research questions with "old" data. It is…
Background. Reproducibility is essential to the scientific method, but reproduction is often a laborious task. Recent works have attempted to automate this process and relieve researchers of this workload. However, due to varying…
The software engineering research community is productive, yet it faces a constellation of challenges: swamped review processes, metric-driven incentives, distorted publication practices, and increasing pressures from AI, scale, and…