Related papers: DecentPeeR: A Self-Incentivised & Inclusive Decent…
Peer review is a widely utilized feedback mechanism for engaging students. As a pedagogical method, it has been shown to improve educational outcomes, but we have found limited empirical measurement of peer review in visualization courses.…
The computing research community needs to work much harder to address the downsides of our innovations. Between the erosion of privacy, threats to democracy, and automation's effect on employment (among many other issues), we can no longer…
The increasing scale and variability of peer review in scholarly venues has created an urgent need for systematic, interpretable, and extensible tools to assess review quality. We present PeeriScope, a modular platform that integrates…
All proper scoring rules incentivize an expert to predict \emph{accurately} (report their true estimate), but not all proper scoring rules equally incentivize \emph{precision}. Rather than treating the expert's belief as exogenously given,…
I propose the R-Index, defined as the difference between the sum of review responsibilities for a researcher's publications and the number of reviews they have completed, as a novel metric to effectively characterize a researcher's…
Peer-to-Peer networks are designed to rely on resources of their own users. Therefore, resource management plays an important role in P2P protocols. Therefore, resource management plays an important role in P2P protocols. Early P2P networks…
The principle of peer review is central to the evaluation of research, by ensuring that only high-quality items are funded or published. But peer review has also received criticism, as the selection of reviewers may introduce biases in the…
In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. In this work, we aim at introducing a methodology to explore whether citation-based metrics, calculated…
The number of citations received by authors in scientific journals has become a major parameter to assess individual researchers and the journals themselves through the impact factor. A fair assessment therefore requires that the criteria…
This essay delves into the ethical dilemmas encountered within the academic peer review process and investigates the prevailing deficiencies in this system. It highlights how established scholars often adhere to mainstream theories not out…
Researchers are often evaluated by citation-based metrics. Such metrics can inform hiring, promotion, and funding decisions. Concerns have been expressed that popular citation-based metrics incentivize researchers to maximize the production…
The advancement of various fields of science depends on the actions of individual scientists via the peer review process. The referees' work patterns and stochastic nature of decision making both relate to the particular features of…
Generative AI (GenAI) use in research writing is growing fast. However, it is unclear how peer reviewers recognize or misjudge AI-augmented manuscripts. To investigate the impact of AI-augmented writing on peer reviews, we conducted a…
Peer review is a widely accepted mechanism for research evaluation, playing a pivotal role in academic publishing. However, criticisms have long been leveled at this mechanism, mostly because of its poor efficiency and low reproducibility.…
The article focuses on the importance of the automatic assignment of reviewers to papers for increasing the assignment accuracy therefore the quality of the scientific event itself. It discusses the main aspects that influence the…
Language models are capable of iteratively improving their outputs based on natural language feedback, thus enabling in-context optimization of user preference. In place of human users, a second language model can be used as an evaluator,…
Despite peer-reviewing being an essential component of academia since the 1600s, it has repeatedly received criticisms for lack of transparency and consistency. We posit that recent work in machine learning and explainable AI provide tools…
Large conferences such as NeurIPS and AAAI serve as crossroads of various AI fields, since they attract submissions from a vast number of communities. However, in some cases, this has resulted in a poor reviewing experience for some…
A method is presented for evaluating authors on the basis of citations. It assigns to each author a citation score which depends upon the number of times he is cited, and upon the scores of the citers. The scores are found to be the…
Recommender systems have become the dominant means of curating cultural content, significantly influencing individual cultural experience. Since recommender systems tend to optimize for personalized user experience, they can overlook…