Related papers: Contrasting Probabilistic Scoring Rules

Virtually all machine learning tasks are characterized using some form of loss function, and "good performance" is typically stated in terms of a sufficiently small average loss, taken over the random draw of test data. While optimizing for…

We consider a simple model of imprecise comparisons: there exists some $\delta>0$ such that when a subject is given two elements to compare, if the values of those elements (as perceived by the subject) differ by at least $\delta$, then the…

When optimizing problems with uncertain parameter values in a linear objective, decision-focused learning enables end-to-end learning of these values. We are interested in a stochastic scheduling problem, in which processing times are…

Decision markets are mechanisms for selecting one among a set of actions based on forecasts about their consequences. Decision markets that are based on scoring rules have been proven to offer incentive compatibility analogous to properly…

Binary observations are often repeated to improve data quality, creating technical replicates. Several scoring methods are commonly used to infer the actual individual state and obtain a probability for each state. The common practice of…

Determination of the range of a variety of social choice correspondences: Plurality voting, the Borda rule, the Pareto rule, the Copeland correspondence, approval voting, and the top cycle correspondence

IRT models are being increasingly used worldwide for test construction and scoring. The study examines the practical implications of estimating individual scores in a paper-and-pencil high-stakes test using 2PL and 3PL models, specifically…

Spearman's rank correlation test is commonly used in astronomy to discern whether a set of two variables are correlated or not. Unlike most other quantities quoted in astronomical literature, the Spearman's rank correlation coefficient is…

Considering voting rules based on evaluation inputs rather than preference rankings modifies the paradigm of probabilistic studies of voting procedures. This article proposes several simulation models for generating evaluation-based voting…

Comparison data arises in many important contexts, e.g. shopping, web clicks, or sports competitions. Typically we are given a dataset of comparisons and wish to train a model to make predictions about the outcome of unseen comparisons. In…

We consider different choice procedures such as scoring rules, rules, using majority relation, value function and tournament matrix, which are used in social and multi-criteria choice problems. We focus on the study of the properties that…

Ensemble forecasting of nonlinear systems involves the use of a model to run forward a discrete ensemble (or set) of initial states. Data assimilation techniques tend to focus on estimating the true state of the system, even though model…

We investigate differences between a simple Dominance Principle applied to sums of fair prices for variables and dominance applied to sums of forecasts for variables scored by proper scoring rules. In particular, we consider differences…

Common machine learning settings range from supervised tasks, where accurately labeled data is accessible, through semi-supervised and weakly-supervised tasks, where target labels are scant or noisy, to unsupervised tasks where labels are…

Knowing when a classifier's prediction can be trusted is useful in many applications and critical for safely using AI. While the bulk of the effort in machine learning research has been towards improving classifier performance,…

Lasso and other regularization procedures are attractive methods for variable selection, subject to a proper choice of shrinkage parameter. Given a set of potential subsets produced by a regularization algorithm, a consistent model…

Voting rules based on evaluation inputs rather than preference orders have been recently proposed, like majority judgement, range voting or approval voting. Traditionally, probabilistic analysis of voting rules supposes the use of…

How predictable a word is can be quantified in two ways: using human responses to the cloze task or using probabilities from language models (LMs).When used as predictors of processing effort, LM probabilities outperform probabilities…

Prediction markets are well-studied in the case where predictions are probabilities or expectations of future random variables. In 2008, Lambert, et al. proposed a generalization, which we call "scoring rule markets" (SRMs), in which…

This study conducts a benchmarking study, comparing 23 different statistical and machine learning methods in a credit scoring application. In order to do so, the models' performance is evaluated over four different data sets in combination…