Related papers: Tournament-Based Performance Evaluation and System…
The evaluation of general-purpose artificial agents, particularly those based on LLMs, presents a significant challenge due to the non-transitive nature of their interactions. When agent A defeats B, B defeats C, and C defeats A,…
Many multi-agent coordination problems can be represented as DCOPs. Motivated by task allocation in disaster response, we extend standard DCOP models to consider uncertain task rewards where the outcome of completing a task depends on its…
Ranking is fundamental to many areas, such as search engine optimization, human feedback for language models, as well as peer grading. Crowdsourcing, which is often used for these tasks, requires proper incentivization to ensure accurate…
A misspecified reward can degrade sample efficiency and induce undesired behaviors in reinforcement learning (RL) problems. We propose symbolic reward machines for incorporating high-level task knowledge when specifying the reward signals.…
Reward models (RMs) play a crucial role in aligning large language models (LLMs) with human preferences and enhancing reasoning quality. Traditionally, RMs are trained to rank candidate outputs based on their correctness and coherence.…
This paper provides a novel solution to a task allocation problem, by which a group of agents decides on the assignment of a discrete set of tasks in a distributed manner. In this setting, heterogeneous agents have individual preferences…
The group draw of a sports tournament requires assigning teams to groups of (almost) the same size. The most important criteria for a draw procedure are balance, randomness, and transparency, which could not be satisfied simultaneously if…
Reinforcement learning (RL) systems can be complex and non-interpretable, making it challenging for non-AI experts to understand or intervene in their decisions. This is due in part to the sequential nature of RL in which actions are chosen…
In fantasy sports, strategic thinking-not mere luck-often defines who wins and who falls short. As fantasy cricket grows in popularity across India, understanding whether success stems from skill or chance has become both an analytical and…
The distribution of efficient individuals in the economy and the efforts that they will put in if they are hired, there are two important concerns for a technologically advanced firm. wants to open a new branch. The firm does not have…
This paper studies a dynamic discrete-time queuing model where at every period players get a new job and must send all their jobs to a queue that has a limited capacity. Players have an incentive to send their jobs as late as possible;…
Over the past two decades, the notion of implicit bias has come to serve as an important component in our understanding of discrimination in activities such as hiring, promotion, and school admissions. Research on implicit bias posits that…
The values of software organizations are crucial for achieving high performance; in particular, agile development approaches emphasize their importance. Researchers have thus far often assumed that a specific set of values, compatible with…
We introduce a new measure to capture fairness of a schedule in a single round robin (SRR) tournament when participants are ranked by strength. To prevent distortion of the outcome of an SRR tournament as well as to guarantee equal…
Ladder tournaments are widely used to rank individuals in real-world organizations and games. Their mathematical properties however are still poorly understood. We formalize the ranking rule generated by a ladder tournament, and we show…
As intelligent agents become more generally-capable, i.e. able to master a wide variety of tasks, the complexity and cost of properly evaluating them rises significantly. Tasks that assess specific capabilities of the agents can be…
The use of algorithmic decision making systems in domains which impact the financial, social, and political well-being of people has created a demand for these decision making systems to be "fair" under some accepted notion of equity. This…
In many social computing applications such as online Q&A forums, the best contribution for each task receives some high reward, while all remaining contributions receive an identical, lower reward irrespective of their actual qualities.…
Quality diversity (QD) is a branch of evolutionary computation that seeks high-quality and behaviorally diverse solutions to a problem. While adversarial problems are common, classical QD cannot be easily applied to them, as both the…
Rankings are central to decision-making in fields ranging from education to online platforms, yet classical deterministic methods such as the Borda count method or Copeland-type pairwise methods ignore uncertainty due to sampling noise or…