相关论文: Do peers share the same criteria for assessing gra…
Many previous causal inference studies require no interference, that is, the potential outcomes of a unit do not depend on the treatments of other units. However, this no-interference assumption becomes unreasonable when a unit interacts…
Background: One of the most important current challenges of Software Engineering (SE) research is to provide relevant evidence to practice. In health related fields, Rapid Reviews (RRs) have shown to be an effective method to achieve that…
Does past performance influence success in grant applications? In this study we test whether the grant allocation decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of the…
Individual performance and reputation within a company are major factors that influence wage distribution, promotion and firing. Due to the complexity and collaborative nature of contemporary business processes, the evaluation of individual…
Conventional collaborative filtering techniques don't take into consideration the effect of discrepancy in users' rating perception. Some users may rarely give 5 stars to items while others almost always assign 5 stars to the chosen item.…
As performance gains through scaling data and/or model size experience diminishing returns, it is becoming increasingly popular to turn to ensembling, where the predictions of multiple models are combined to improve accuracy. In this paper,…
Existing interpretation algorithms have found that, even deep models make the same and right predictions on the same image, they might rely on different sets of input features for classification. However, among these sets of features, some…
Accurately grading open-ended assignments in large or massive open online courses (MOOCs) is non-trivial. Peer review is a promising solution but can be unreliable due to few reviewers and an unevaluated review form. To date, no work has 1)…
Providing consistent summative assessment to students is important, as the grades they are awarded affect their progression through university and future career prospects. While small cohorts are typically assessed by a single assessor,…
The "free rider" problem has long plagued pedagogies based on collaborative learning. The most common solution to the free rider problem is peer evaluation. As well other existing methods of peer evaluation include self-evaluation --- and…
Peer review is crucial for advancing and improving science through constructive criticism. However, toxic feedback can discourage authors and hinder scientific progress. This work explores an important but underexplored area: detecting…
Academic publishing requires solving a collective coordination problem: among thousands of possible publication venues, which deserve a community's attention? A clear consensus helps scholars allocate attention, match submissions to…
Recent work in large language modeling (LLMs) has used fine-tuning to align outputs with the preferences of a prototypical user. This work assumes that human preferences are static and homogeneous across individuals, so that aligning to a a…
This paper studies preference aggregation under risk. In our model, each agent has an incomplete preference relation represented by a set of expected utility functions. The classical Pareto principle is silent on agreement involving…
As the final stage of recommender systems, re-ranking presents ordered item lists to users that best match their interests. It plays such a critical role and has become a trending research topic with much attention from both academia and…
Background: When selecting predictive tools, for implementation in clinical practice or for recommendation in guidelines, clinicians are challenged with an overwhelming and ever-growing number of tools. Many of these have never been…
There is widespread debate on whether to anonymize author identities in peer review. The key argument for anonymization is to mitigate bias, whereas arguments against anonymization posit various uses of author identities in the review…
Effective peer assessment requires students to be attentive to the deficiencies in the work they rate. Thus, their reviews should identify problems. But what ways are there to check that they do? We attempt to automate the process of…
Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high…
Pairwise comparisons are a well-known method for the representation of the subjective preferences of a decision maker. Evaluating their inconsistency has been a widely studied and discussed topic and several indices have been proposed in…