Related papers: Crowd IQ -- Aggregating Opinions to Boost Performa…
Respecting minority opinions is vital in solving social problems. However, minority opinions are often ignored in general majority rules. To build consensus on pluralistic values and make social choices that consider minority opinions, we…
Expertise of annotators has a major role in crowdsourcing based opinion aggregation models. In such frameworks, accuracy and biasness of annotators are occasionally taken as important features and based on them priority of the annotators…
Crowdsourcing offers a practical method for ranking and scoring large amounts of items. To investigate the algorithms and incentives that can be used in crowdsourcing quality evaluations, we built CrowdGrader, a tool that lets students…
Crowdsourcing is a process of accumulating the ideas, thoughts or information from many independent participants, with aim to find the best solution for a given challenge. Modern information technologies allow for massive number of subjects…
Crowdsourcing allows to instantly recruit workers on the web to annotate image, web page, or document databases. However, worker unreliability prevents taking a workers responses at face value. Thus, responses from multiple workers are…
Crowdsourcing can identify high-quality solutions to problems; however, individual decisions are constrained by cognitive biases. We investigate some of these biases in an experimental model of a question-answering system. In both natural…
We study a model of a population making a binary decision based on information spreading within the population, which is fully connected or covering a square grid. We assume that a fraction of the population wants to make the choice of the…
Reaching some form of consensus is often necessary for autonomous agents that want to coordinate their actions or otherwise engage in joint activities. One way to reach a consensus is by aggregating individual information, such as…
Crowdsourcing information constitutes an important aspect of human-in-the-loop learning for researchers across multiple disciplines such as AI, HCI, and social science. While using crowdsourced data for subjective tasks is not new,…
In our digital and connected societies, the development of social networks, online shopping, and reputation systems raises the question of how individuals use social information, and how it affects their decisions. We report experiments…
The increasing practice of engaging crowds, where organizations use IT to connect with dispersed individuals for explicit resource creation purposes, has precipitated the need to measure the precise processes and benefits of these…
Aggregating responses from crowd workers is a fundamental task in the process of crowdsourcing. In cases where a few experts are overwhelmed by a large number of non-experts, most answer aggregation algorithms such as the majority voting…
Crowdsourcing offers an affordable and scalable means to collect relevance judgments for IR test collections. However, crowd assessors may show higher variance in judgment quality than trusted assessors. In this paper, we investigate how to…
As acquiring reliable ground-truth labels is usually costly, or infeasible, crowdsourcing and aggregation of noisy human annotations is the typical resort. Aggregating subjective labels, though, may amplify individual biases, particularly…
Learning from the crowd has become increasingly popular in the Web and social media. There is a wide variety of crowdlearning sites in which, on the one hand, users learn from the knowledge that other users contribute to the site, and, on…
Crowd predictions have demonstrated powerful performance in predicting future events. We aim to understand crowd prediction efficacy in ascertaining the veracity of human emotional expressions. We discover that collective discernment can…
Cognitive biases are widespread in humans and animals alike, and can sometimes be reinforced by social interactions. One prime bias in judgment and decision-making is the human tendency to underestimate large quantities. Previous research…
We propose a simple model to explore an educational phenomenon where the correct answer emerges from group discussion. We construct our model based on several plausible assumptions: (i) We tend to follow peers' opinions. However, if a…
Crowdsourcing has been widely used to efficiently obtain labeled datasets for supervised learning from large numbers of human resources at low cost. However, one of the technical challenges in obtaining high-quality results from…
Wisdom of crowds refers to the phenomenon that the aggregate prediction or forecast of a group of individuals can be surprisingly more accurate than most individuals in the group, and sometimes - than any of the individuals comprising it.…