Related papers: Analyzing the Effects of Two-Stage Peer Evaluation
In many settings, an effective way of evaluating objects of interest is to collect evaluations from dispersed individuals and to aggregate these evaluations together. Some examples are categorizing online content and evaluating student…
Many conferences rely on paper bidding as a key component of their reviewer assignment procedure. These bids are then taken into account when assigning reviewers to help ensure that each reviewer is assigned to suitable papers. However,…
While lobbying has been demonstrated to have an important effect on public opinion and policy making, existing models of opinion formation do not specifically include its effect. In this work we introduce a new model of lobbying-driven…
We investigate a social system of agents faced with a binary choice. We assume there is a correct, or beneficial, outcome of this choice. Furthermore, we assume agents are influenced by others in making their decision, and that the agents…
Modern AI systems increasingly operate inside markets and institutions where data, behavior, and incentives are endogenous. This paper develops an economic foundation for multi-agent learning by studying a principal-agent interaction in a…
Peer review is the main mechanism by which the software engineering community assesses the quality of scientific results. However, the rapid growth of paper submissions in software engineering venues has outpaced the availability of…
Many previous causal inference studies require no interference, that is, the potential outcomes of a unit do not depend on the treatments of other units. However, this no-interference assumption becomes unreasonable when a unit interacts…
Peer review is the method employed by the scientific community for evaluating research advancements. In the field of cybersecurity, the practice of double-blind peer review is the de-facto standard. This paper touches on the holy grail of…
This paper considers the scenario in which there are multiple institutions, each with a limited capacity for candidates, and candidates, each with preferences over the institutions. A central entity evaluates the utility of each candidate…
Counterfactual learning to rank (CLTR) aims to learn a ranking policy from user interactions while correcting for the inherent biases in interaction data, such as position bias. Existing CLTR methods assume a single ranking policy that…
We consider a distributed voting problem with a set of agents that are partitioned into disjoint groups and a set of obnoxious alternatives. Agents and alternatives are represented by points in a metric space. The goal is to compute the…
We use variation of test scores measuring closely related skills to isolate peer effects. The intuition for our identification strategy is that the difference in closely related scores eliminates factors common to the performance in either…
We analyze Assessment Voting, a new two-round voting procedure that can be applied to binary decisions in democratic societies. In the first round, a randomly-selected number of citizens cast their vote on one of the two alternatives at…
In this paper, we introduce a two-stage Bayesian persuasion model in which a third-party platform controls the information available to the sender about users' preferences. We aim to characterize the optimal information disclosure policy of…
In policy learning, the goal is typically to optimize a primary performance metric, but other subsidiary metrics often also warrant attention. This paper presents two strategies for evaluating these subsidiary metrics under a policy that is…
The evaluation of recommendation systems is a complex task. The offline and online evaluation metrics for recommender systems are ambiguous in their true objectives. The majority of recently published papers benchmark their methods using…
As AI research surges in both impact and volume, conferences have imposed submission limits to maintain paper quality and alleviate organizational pressure. In this work, we examine the fairness of desk-rejection systems under submission…
Peer review is an essential process to determine the quality of papers submitted to scientific conferences or journals. However, it is subjective and prone to biases. Several studies have been conducted to apply techniques from NLP to…
In many institutional settings, $k$ items are selected with the goal of representing the underlying distribution of claims, opinions, or characteristics in a large population. We study environments with two adversarial parties whose…
As AI usage becomes more prevalent in social contexts, understanding agent-user interaction is critical to designing systems that improve both individual and group outcomes. We present an online behavioral experiment (N = 243) in which…