Related papers: Elicitability
Incorporation of expert information in inference or decision settings is often important, especially in cases where data are unavailable, costly or unreliable. One approach is to elicit prior quantiles from an expert and then to fit these…
Many experiments elicit subjects' prior and posterior beliefs about a random variable to assess how information affects one's own actions. However, beliefs are multi-dimensional objects, and experimenters often only elicit a single response…
A crucial part of data analysis is the validation of the resulting estimators, in particular, if several competing estimators need to be compared. Whether an estimator can be objectively validated is not a trivial property. If there exists…
It is important to collect credible training samples $(x,y)$ for building data-intensive learning systems (e.g., a deep learning system). Asking people to report complex distribution $p(x)$, though theoretically viable, is challenging in…
Reward schemes may affect not only agents' effort, but also their incentives to gather information to reduce the riskiness of the productive activity. In a laboratory experiment using a novel task, we find that the relationship between…
I study whether and which expert incentives can be provided at what cost when the states of the world become non-contractible, but there is some noisy observation about the states that can be contracted upon. A principal hires an agent to…
This article introduces a new method for eliciting prior distributions from experts. The method models an expert decision-making process to infer a prior probability distribution for a rare event $A$. More specifically, assuming there…
Comparison data elicited from people are fundamental to many machine learning tasks, including reinforcement learning from human feedback for large language models and estimating ranking models. They are typically subjective and not…
Belief elicitation is ubiquitous in experiments but can distort behavior in the main tasks. We study when, and how, an experimenter can ask for a series of action-dependent belief statistics after a subject chooses an action, while…
Elicitability is a property of $\mathbb{R}^k$-valued functionals defined on a set of distribution functions. These functionals represent statistical properties of a distribution, for instance its mean, variance, or median. They are called…
We consider active learning under incentive compatibility constraints. The main application of our results is to economic experiments, in which a learner seeks to infer the parameters of a subject's preferences: for example their attitudes…
In 1977 John Tukey described how in exploratory data analysis, data analysts use tools, such as data visualizations, to separate their expectations from what they observe. In contrast to statistical theory, an underappreciated aspect of…
In crowdsourcing when there is a lack of verification for contributed answers, output agreement mechanisms are often used to incentivize participants to provide truthful answers when the correct answer is hold by the majority. In this…
In many settings -- like market research and social choice -- people may be presented with unfamiliar options. Classical mechanisms may perform poorly because they fail to incentivize people to learn about these options, or worse, encourage…
For obtaining causal inferences that are objective, and therefore have the best chance of revealing scientific truths, carefully designed and executed randomized experiments are generally considered to be the gold standard. Observational…
We consider the design of experiments to evaluate treatments that are administered by self-interested agents, each seeking to achieve the highest evaluation and win the experiment. For example, in an advertising experiment, a company wishes…
Connections appear to be helpful in many contexts, such as obtaining a job, a promotion, a grant, a loan, or publishing a paper. This may be due either to favoritism or to information conveyed by connections. Attempts at identifying both…
The publication process both determines which research receives the most attention, and influences the supply of research through its impact on researchers' private incentives. We introduce a framework to study optimal publication decisions…
Common sense suggests that when individuals explain why they believe something, we can arrive at more accurate conclusions than when they simply state what they believe. Yet, there is no known mechanism that provides incentives to elicit…
How can one efficiently share payoffs with collaborators when participating in risky research? First, I show that efficiency can be achieved by allocating payoffs asymmetrically between the researcher who makes a breakthrough ("winner") and…