Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms
Abstract
Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriously inflated and uninformative. In this paper, we draw inspiration from historical worker guilds (e.g., in the silk trade) to design and implement crowd guilds: centralized groups of crowd workers who collectively certify each other's quality through double-blind peer assessment. A two-week field experiment compared crowd guilds to a traditional decentralized crowd work model. Crowd guilds produced reputation signals more strongly correlated with ground-truth worker quality than signals available on current crowd working platforms, and more accurate than in the traditional model.
Keywords
Cite
@article{arxiv.1611.01572,
title = {Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms},
author = {Mark E. Whiting and Dilrukshi Gamage and Snehalkumar S. Gaikwad and Aaron Gilbee and Shirish Goyal and Alipta Ballav and Dinesh Majeti and Nalin Chhibber and Angela Richmond-Fuller and Freddie Vargus and Tejas Seshadri Sarma and Varshine Chandrakanthan and Teogenes Moura and Mohamed Hashim Salih and Gabriel Bayomi Tinoco Kalejaiye and Adam Ginzberg and Catherine A. Mullings and Yoni Dayan and Kristy Milland and Henrique Orefice and Jeff Regino and Sayna Parsi and Kunz Mainali and Vibhor Sehgal and Sekandar Matin and Akshansh Sinha and Rajan Vaish and Michael S. Bernstein},
journal= {arXiv preprint arXiv:1611.01572},
year = {2020}
}
Comments
12 pages, 6 figures, 1 table. To be presented at CSCW2017