Related papers: Paper Quality Assessment based on Individual Wisdo…
Is it possible to reliably evaluate the quality of peer reviews? We study this question driven by two primary motivations -- incentivizing high-quality reviewing using assessed quality of reviews and measuring changes to review quality in…
Peer review is the primary mechanism for evaluating scientific contributions, yet prior studies have mostly examined paper features or external metadata in isolation. The emergence of open platforms such as OpenReview has transformed peer…
One of the virtues of peer review is that it provides a self-regulating selection mechanism for scientific work, papers and projects. Peer review as a selection mechanism is hard to evaluate in terms of its efficiency. Serious efforts to…
Scholarly journals rely on peer review to identify the science most worthy of publication. Yet finding willing and qualified reviewers to evaluate manuscripts has become an increasingly challenging task, possibly even threatening the…
Peer review shapes which scientific claims enter the published record, but its internal dynamics are hard to measure at scale because reviewer criticism and author revision are usually embedded in long, unstructured correspondence. Here we…
Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of…
Peer review is a laborious, yet essential, part of academic publishing with crucial impact on the scientific endeavor. The current lack of incentives and transparency harms the credibility of this process. Researchers are neither rewarded…
The principle of peer review is central to the evaluation of research, by ensuring that only high-quality items are funded or published. But peer review has also received criticism, as the selection of reviewers may introduce biases in the…
The number of scientific publications is constantly rising, increasing the strain on the review process. The number of submissions is actually higher, as each manuscript is often reviewed several times before publication. To face the deluge…
Peer review is widely regarded as essential for advancing scientific research. However, reviewers may be biased by authors' prestige or other characteristics. Double-blind peer review, in which the authors' identities are masked from the…
We develop a simple model of the scientific peer review process, in which authors of varying ability invest to produce papers of varying quality, and journals evaluate papers based on a noisy signal, choosing to accept or reject each paper.…
Scientific publishing is in a transition between the old paper-bound, static forms and the new electronic media with its interactive, dynamic possibilities. This takes place in the context of imploding library budgets and exploding magazine…
When eliciting opinions from a group of experts, traditional devices used to promote honest reporting assume that there is an observable future outcome. In practice, however, this assumption is not always reasonable. In this paper, we…
Peer assessment has established itself as a critical pedagogical tool in academic settings, offering students timely, high-quality feedback to enhance learning outcomes. However, the efficacy of this approach depends on two factors: (1) the…
Scientific evaluation is a determinant of how scientists, institutions and funders behave, and as such is a key element in the making of science. In this article, we propose an alternative to the current norm of evaluating research with…
Peer review is a cornerstone of scientific publishing, including at premier machine learning conferences such as ICLR. As submission volumes increase, understanding the nature and dynamics of the review process is crucial for improving its…
In computer science, conferences and journals conduct peer review in order to decide what to publish. Many have pointed out the inherent weaknesses in peer review, including those of bias, quality, and accountability. Many have suggested…
In this paper peer review reliability is investigated based on peer ratings of research teams at two Belgian universities. It is found that outcomes can be substantially influenced by the different ways in which experts attribute ratings.…
Despite peer-reviewing being an essential component of academia since the 1600s, it has repeatedly received criticisms for lack of transparency and consistency. We posit that recent work in machine learning and explainable AI provide tools…
We propose an enhanced peer-review process where the reviewers are encouraged to truthfully disclose their reviews. We start by modelling that process using a Bayesian model where the uncertainty regarding the quality of the manuscript is…