Related papers: On a Reliable Peer-Review Process
When eliciting opinions from a group of experts, traditional devices used to promote honest reporting assume that there is an observable future outcome. In practice, however, this assumption is not always reasonable. In this paper, we…
Peer grading systems aggregate noisy reports from multiple students to approximate a true grade as closely as possible. Most current systems either take the mean or median of reported grades; others aim to estimate students' grading…
The peer-review process is the most widely accepted certification mechanism for officially accepting the written results of researchers within the scientific community. An essential component of peer-review is the identification of…
Traditional closed peer review systems, which have played a central role in scientific publishing, are often slow, costly, non-transparent, stochastic, and possibly subject to biases - factors that can impede scientific progress and…
Scholarly journals rely on peer review to identify the science most worthy of publication. Yet finding willing and qualified reviewers to evaluate manuscripts has become an increasingly challenging task, possibly even threatening the…
We develop a simple model of the scientific peer review process, in which authors of varying ability invest to produce papers of varying quality, and journals evaluate papers based on a noisy signal, choosing to accept or reject each paper.…
Is it possible to reliably evaluate the quality of peer reviews? We study this question driven by two primary motivations -- incentivizing high-quality reviewing using assessed quality of reviews and measuring changes to review quality in…
We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy. Our fairness objective is to maximize the review quality of the most disadvantaged paper,…
A successful peer review process requires that qualified and interested reviewers are assigned to each paper. Most automated reviewer assignment approaches estimate a real-valued affinity score for each paper-reviewer pair that acts as a…
The number of scientific publications is constantly rising, increasing the strain on the review process. The number of submissions is actually higher, as each manuscript is often reviewed several times before publication. To face the deluge…
In computer science, conferences and journals conduct peer review in order to decide what to publish. Many have pointed out the inherent weaknesses in peer review, including those of bias, quality, and accountability. Many have suggested…
Peer review is a laborious, yet essential, part of academic publishing with crucial impact on the scientific endeavor. The current lack of incentives and transparency harms the credibility of this process. Researchers are neither rewarded…
We study a problem where a group of agents has to decide how a joint reward should be shared among them. We focus on settings where the share that each agent receives depends on the subjective opinions of its peers concerning that agent's…
The importance of peer-review in the scientific process can not be overestimated. Yet, due to increasing pressures of research and exponentially growing number of publications the task faced by the referees becomes ever more difficult. We…
Peer grading systems work well only if users have incentives to grade truthfully. An example of non-truthful grading, that we observed in classrooms, consists in students assigning the maximum grade to all submissions. With a naive grading…
Peer review, as a widely used practice to ensure the quality and integrity of publications, lacks a well-defined and common mechanism to self-incentivize virtuous behavior across all the conferences and journals. This is because information…
In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the isotonic mechanism, which can elicit the ranking of paper…
In this paper peer review reliability is investigated based on peer ratings of research teams at two Belgian universities. It is found that outcomes can be substantially influenced by the different ways in which experts attribute ratings.…
Peer-prediction is a (meta-)mechanism which, given any proper scoring rule, produces a mechanism to elicit privately-held, non-verifiable information from self-interested agents. Formally, truth-telling is a strict Nash equilibrium of the…
Peer assessment has established itself as a critical pedagogical tool in academic settings, offering students timely, high-quality feedback to enhance learning outcomes. However, the efficacy of this approach depends on two factors: (1) the…