Cheaper and Better: Selecting Good Workers for Crowdsourcing
Abstract
Crowdsourcing provides a popular paradigm for data collection at scale. We study the problem of selecting subsets of workers from a given worker pool to maximize the accuracy under a budget constraint. One natural question is whether we should hire as many workers as the budget allows, or restrict on a small number of top-quality workers. By theoretically analyzing the error rate of a typical setting in crowdsourcing, we frame the worker selection problem into a combinatorial optimization problem and propose an algorithm to solve it efficiently. Empirical results on both simulated and real-world datasets show that our algorithm is able to select a small number of high-quality workers, and performs as good as, sometimes even better than, the much larger crowds as the budget allows.
Cite
@article{arxiv.1502.00725,
title = {Cheaper and Better: Selecting Good Workers for Crowdsourcing},
author = {Hongwei Li and Qiang Liu},
journal= {arXiv preprint arXiv:1502.00725},
year = {2015}
}