相关论文: A System-Level Analysis of Conference Peer Review
Peer review is a key component of the publishing process in most fields of science. The increasing submission rates put a strain on reviewing quality and efficiency, motivating the development of applications to support the reviewing and…
Peer grading systems make large courses more scalable, provide students with faster and more detailed feedback, and help students to learn by thinking critically about the work of others. A key obstacle to the broader adoption of peer…
Large language models offer a tempting solution to address the peer review crisis. This position paper argues that today's AI systems should not be used to produce paper reviews. We ground this position in an empirical comparison of human-…
Many governments have placed priority on excellence in higher education as part of their policy agendas. Processes for recruitment and career advancement in universities thus have a critical role. The efficiency of faculty selection…
Peer selection, the evaluation and selection of agents by their peers, is an important problem in the field of computational social choice; with applications to grading in massively online courses (MOOCs) and academic peer review. Current…
Devising an appropriate scheme that assigns the weights to share credits among multiple authors of a paper is a challenging task. This challenge comes from the fact that different types of conventions might be followed among different…
We study how the design of admissions policies affects the ability of students admitted to universities. In our model, applicants have a multi-dimensional ability, which is a combination of a "type" and a "soft skill." Universities may…
The adoption of large language models (LLMs) is transforming the peer review process, from assisting reviewers in writing detailed evaluations to generating entire reviews automatically. While these capabilities offer new opportunities,…
A simulation model based on parallel systems is established, aiming to explore the relation between the number of submissions and the overall standard of academic journals within a similar discipline under peer review. The model can…
Peer reviewing is a central process in modern research and essential for ensuring high quality and reliability of published work. At the same time, it is a time-consuming process and increasing interest in emerging fields often results in a…
Top-tier academic conferences are failing under the strain of two irreconcilable roles: (1) rapid dissemination of all sound research and (2) scarce credentialing for prestige and career advancement. This conflict has created a reviewer…
We present the NeurIPS 2021 consistency experiment, a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process.…
Peer review is our best tool for judging the quality of conference submissions, but it is becoming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing…
We present an online deliberation system using mutual evaluation in order to collaboratively develop solutions. Participants submit their proposals and evaluate each other's proposals; some of them may then be invited by the system to…
Generative artificial intelligence (AI) has begun to reshape academic publishing by enabling the rapid production of submission-ready manuscripts. While such tools promise to enhance productivity, they also raise concerns about overwhelming…
A `peer-review system' in the context of judging research contributions, is one of the prime steps undertaken to ensure the quality of the submissions received, a significant portion of the publishing budget is spent towards successful…
The increasing volume of scientific publications and grant proposals has generated an unprecedentedly high workload to scientific communities. Consequently, review quality has been decreasing and review outcomes have become less correlated…
In this paper, we navigate the intricate domain of reviewer rewards in open-access academic publishing, leveraging the precision of mathematics and the strategic acumen of game theory. We conceptualize the prevailing voucher-based reviewer…
The increasing scale and variability of peer review in scholarly venues has created an urgent need for systematic, interpretable, and extensible tools to assess review quality. We present PeeriScope, a modular platform that integrates…
Machine learning and artificial intelligence conferences such as NeurIPS and ICML now regularly receive tens of thousands of submissions, posing significant challenges to maintaining the quality and consistency of the peer review process.…