Related papers: Elicitation Complexity of Statistical Properties
Property elicitation studies which attributes of a probability distribution can be determined by minimizing a risk. We investigate a generalization of property elicitation to imprecise probabilities (IP). This investigation is motivated by…
Elicitability is a property of $\mathbb{R}^k$-valued functionals defined on a set of distribution functions. These functionals represent statistical properties of a distribution, for instance its mean, variance, or median. They are called…
A crucial part of data analysis is the validation of the resulting estimators, in particular, if several competing estimators need to be compared. Whether an estimator can be objectively validated is not a trivial property. If there exists…
A statistical functional, such as the mean or the median, is called elicitable if there is a scoring function or loss function such that the correct forecast of the functional is the unique minimizer of the expected score. Such scoring…
We investigate the application of classification techniques to utility elicitation. In a decision problem, two sets of parameters must generally be elicited: the probabilities and the utilities. While the prior and conditional probabilities…
The risk of a financial position is usually summarized by a risk measure. As this risk measure has to be estimated from historical data, it is important to be able to verify and compare competing estimation procedures. In statistical…
What is a fair performance metric? We consider the choice of fairness metrics through the lens of metric elicitation -- a principled framework for selecting performance metrics that best reflect implicit preferences. The use of metric…
Complexity theory is a useful tool to study computational issues surrounding the elicitation of preferences, as well as the strategic manipulation of elections aggregating together preferences of multiple agents. We study here the…
Metric elicitation is a recent framework for eliciting classification performance metrics that best reflect implicit user preferences based on the task and context. However, available elicitation strategies have been limited to linear (or…
Given a learning problem with real-world tradeoffs, which cost function should the model be trained to optimize? This is the metric selection problem in machine learning. Despite its practical interest, there is limited formal guidance on…
We provide a constructive way of defining new elicitable risk measures that are characterised by a multiplicative scoring function. We show that depending on the choice of the scoring function's components, the resulting risk measure…
One prominent method of evaluating machine learning model trustworthiness is the notion of calibration. In the binary outcome setting, a probabilistic predictor is calibrated if outcomes are realized according to a model's distributional…
Established methods for structural elicitation typically rely on code modelling standard graphical models classes, most often Bayesian networks. However, more appropriate models may arise from asking the expert questions in common language…
We study loss functions that measure the accuracy of a prediction based on multiple data points simultaneously. To our knowledge, such loss functions have not been studied before in the area of property elicitation or in machine learning…
A good process model is expected not only to reflect the behavior of the process, but also to be as easy to read and understand as possible. Because preferences vary across different applications, numerous measures provide ways to reflect…
In building Bayesian belief networks, the elicitation of all probabilities required can be a major obstacle. We learned the extent of this often-cited observation in the construction of the probabilistic part of a complex influence diagram…
We study the non-parametric isotonic regression problem for bivariate elicitable functionals that are given as an elicitable univariate functional and its Bayes risk. Prominent examples for functionals of this type are (mean, variance) and…
Informally, a risk measure is said to be elicitable if there exists a suitable scoring function such that minimizing its expected value recovers the risk measure. In this paper, we analyze the elicitability properties of the class of return…
A measure of complexity based on a probabilistic description of physical systems is proposed. This measure incorporates the main features of the intuitive notion of such a magnitude. It can be applied to many physical situations and to…
Statistical functionals are called elicitable if there exists a loss or scoring function under which the functional is the optimal point forecast in expectation. While the mean and quantiles are elicitable, it has been shown in Heinrich…