Related papers: Auditing the Auditors: Does Community-based Modera…
Microtask crowdsourcing has enabled dataset advances in social science and machine learning, but existing crowdsourcing schemes are too expensive to scale up with the expanding volume of data. To scale and widen the applicability of…
Opinion dynamics is crucial for unraveling the complexities of human interaction in the information age. How to speed up consensus without disturbing the fate of the system is key for opinion dynamics. We propose a voter model on adaptive…
Large language models show promising capabilities for contextual fact-checking on social media: they can verify contested claims through deep research, synthesize evidence from multiple sources, and draft explanations at scale. However,…
In today's world, social networks have become one of the primary sources for creation and propagation of news. Social news aggregators are one of the actors in this area in which users post news items and use positive or negative votes to…
Crowdsourcing is an easy, cheap, and fast way to perform large scale quality assessment; however, human judgments are often influenced by cognitive biases, which lowers their credibility. In this study, we focus on cognitive biases…
Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy…
Decentralized large language model inference networks require lightweight mechanisms to reward high quality outputs under heterogeneous latency and cost. Proof of Quality provides scalable verification by sampling evaluator nodes that score…
In crowdsourced preference aggregation, it is often assumed that all the annotators are subject to a common preference or utility function which generates their comparison behaviors in experiments. However, in reality annotators are subject…
The increasing reliance on complex algorithmic systems by online platforms has sparked a growing need for algorithm auditing, a methodology evaluating these systems' functionality and impact. In this paper, we systematically review 176…
Corrections given by ordinary social media users, also referred to as Social Correction have emerged as a viable intervention against misinformation as per the recent literature. However, little is known about how often users give disputing…
Blockchains combine other technologies, such as cryptography, networking, and incentive mechanisms, to enable the creation, validation, and recording of transactions between participating nodes. A consensus algorithm is used in a blockchain…
Growing amount of comments make online discussions difficult to moderate by human moderators only. Antisocial behavior is a common occurrence that often discourages other users from participating in discussion. We propose a neural network…
Selecting an effective training signal for machine learning tasks is difficult: expert annotations are expensive, and crowd-sourced annotations may not be reliable. Recent work has demonstrated that learning from a distribution over labels…
Resolving disagreement in manual annotation typically consists of removing unreliable annotators and using a label aggregation strategy such as majority vote or expert opinion to resolve disagreement. These may have the side-effect of…
Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation. Majority vote discards annotator…
A central requirement of the European Union's Digital Services Act (DSA) is that online platforms undergo internal and external audits. A key component of these audits is the assessment of systemic risks, including the dissemination of…
Theoretical work on sequential choice and large-scale experiments in online ranking and voting systems has demonstrated that social influence can have a drastic impact on social and technological systems. Yet, the effect of social influence…
In participatory budgeting we are given a set of projects---each with a cost, an available budget, and a set of voters who in some form express their preferences over the projects. The goal is to select---based on voter preferences---a…
In the era of rapid Internet and social media platform development, individuals readily share their viewpoints online. The overwhelming quantity of these posts renders comprehensive analysis impractical. This necessitates an efficient…
Wiki articles are created and maintained by a crowd of editors, producing a continuous stream of reviews. Reviews can take the form of additions, reverts, or both. This crowdsourcing model is exposed to manipulation since neither reviews…