相关论文: Granular feedback merits sophisticated aggregation

Scientists frequently generalize population level causal quantities such as average treatment effect from a source population to a target population. When the causal effects are heterogeneous, differences in subject characteristics between…

Humans rely more and more on systems with AI components. The AI community typically treats human inputs as a given and optimizes AI models only. This thinking is one-sided and it neglects the fact that humans can learn, too. In this work,…

On the Web, there is always a need to aggregate opinions from the crowd (as in posts, social networks, forums, etc.). Different mechanisms have been implemented to capture these opinions such as "Like" in Facebook, "Favorite" in Twitter,…

We analyze the accuracy of collective decision-making in socially connected populations, where agents update binary choices through local interactions on a network. Each agent receives a private signal that is biased -- even marginally --…

Pretrained language models often do not perform tasks in ways that are in line with our preferences, e.g., generating offensive text or factually incorrect summaries. Recent work approaches the above issue by learning from a simple form of…

In prediction problems, it is common to model the data-generating process and then use a model-based procedure, such as a Bayesian predictive distribution, to quantify uncertainty about the next observation. However, if the posited model is…

Due to the difficulties in replicating and scaling up qualitative studies, such studies are rarely verified. Accordingly, in this paper, we leverage the advantages of crowdsourcing (low costs, fast speed, scalable workforce) to replicate…

Large language models (LLMs) are increasingly deployed via public-facing interfaces to interact with millions of users, each with diverse preferences. Despite this, preference tuning of LLMs predominantly relies on reward models trained…

Crowdsourcing platforms emerged as popular venues for purchasing human intelligence at low cost for large volume of tasks. As many low-paid workers are prone to give noisy answers, a common practice is to add redundancy by assigning…

Crowdsourcing is an easy, cheap, and fast way to perform large scale quality assessment; however, human judgments are often influenced by cognitive biases, which lowers their credibility. In this study, we focus on cognitive biases…

Many recent advances in natural language generation have been fueled by training large language models on internet-scale data. However, this paradigm can lead to models that generate toxic, inaccurate, and unhelpful content, and automatic…

Crowd predictions have demonstrated powerful performance in predicting future events. We aim to understand crowd prediction efficacy in ascertaining the veracity of human emotional expressions. We discover that collective discernment can…

Crowdsourcing enables the solicitation of forecasts on a variety of prediction tasks from distributed groups of people. How to aggregate the solicited forecasts, which may vary in quality, into an accurate final prediction remains a…

The aggregation of many independent estimates can outperform the most accurate individual judgment. This centenarian finding, popularly known as the wisdom of crowds, has been applied to problems ranging from the diagnosis of cancer to…

Complex decision-making systems rarely have direct access to the current state of the world and they instead rely on opinions to form an understanding of what the ground truth could be. Even in problems where experts provide opinions…

Human groups can perform extraordinary accurate estimations compared to individuals by simply using the mean, median or geometric mean of the individual estimations [Galton 1907, Surowiecki 2005, Page 2008]. However, this is true only for…

In this paper, a new randomized response technique aimed at protecting respondents' privacy is proposed. It is designed for estimating the population total, or the population mean, of a quantitative characteristic. It provides a~high degree…

When multiple forecasts are available for a probability distribution, forecast combining enables a pragmatic synthesis of the information to extract the wisdom of the crowd. The linear opinion pool has been widely used, whereby the…

This paper deals with the problem of estimating the finite population mean when some information on two auxiliary attributes are available. It is shown that the proposed estimator is more efficient than the usual mean estimator and other…

Collective intelligence is believed to underly the remarkable success of human society. The formation of accurate shared beliefs is one of the key components of human collective intelligence. How are accurate shared beliefs formed in groups…