Related papers: Ranking earthquake forecasts using proper scoring …
Probability forecasts for binary outcomes, often referred to as probabilistic classifiers or confidence scores, are ubiquitous in science and society, and methods for evaluating and comparing them are in great demand. We propose and study a…
Improvement of the prediction accuracy of the Earth's rotation parameters (ERP) is one of the main problems of applied astrometry. In order to solve this problem, various approaches are used and in order to select the best one, comparison…
Proper scoring rules have been a subject of growing interest in recent years, not only as tools for evaluation of probabilistic forecasts but also as methods for estimating probability distributions. In this article, we review the…
Machine learning models are often used to inform real world risk assessment tasks: predicting consumer default risk, predicting whether a person suffers from a serious illness, or predicting a person's risk to appear in court. Given…
Utilizing established risk factors and prognostic models can often improve the construction of a newer risk model that uses novel biomarkers in a smaller, internal study. However, directly borrowing information from an established…
Comparison data arises in many important contexts, e.g. shopping, web clicks, or sports competitions. Typically we are given a dataset of comparisons and wish to train a model to make predictions about the outcome of unseen comparisons. In…
Probabilistic forecasting in combination with stochastic programming is a key tool for handling the growing uncertainties in future energy systems. Derived from a general stochastic programming formulation for the optimal scheduling and…
The Collaboratory for the Study of Earthquake Predictability (CSEP) aims to prospectively test time-dependent earthquake probability forecasts on their consistency with observations. To compete, time-dependent seismicity models are…
In the practice of point prediction, it is desirable that forecasters receive a directive in the form of a statistical functional, such as the mean or a quantile of the predictive distribution. When evaluating and comparing competing…
We introduce a class of proper scoring rules for evaluating spatial point process forecasts based on summary statistics. These scoring rules rely on Monte-Carlo approximations of expectations and can therefore easily be evaluated for any…
A new notion of stochastic ordering is introduced to compare multivariate stochastic risk models with respect to extreme portfolio losses. In the framework of multivariate regular variation comparison criteria are derived in terms of…
A theoretical analysis of the earthquake prediction problem in space-time is presented. We find an explicit structure of the optimal strategy and its relation to the generalized error diagram. This study is a generalization of the…
Performative predictions are forecasts which influence the outcomes they aim to predict, undermining the existence of correct forecasts and standard methods of elicitation and estimation. We show that conditioning forecasts on covariates…
We introduce a new measure for fair and meaningful comparisons of single-valued output from artificial intelligence based weather prediction (AIWP) and numerical weather prediction (NWP) models, called potential continuous ranked…
Existing metrics in competing risks survival analysis such as concordance and accuracy do not evaluate a model's ability to jointly predict the event type and the event time. To address these limitations, we propose a new metric, which we…
In meteorology, engineering and computer sciences, data assimilation is routinely employed as the optimal way to combine noisy observations with prior model information for obtaining better estimates of a state, and thus better forecasts,…
Ranks estimated from data are uncertain and this poses a challenge in many applications. However, estimated ranks are deterministic functions of estimated parameters, so the uncertainty in the ranks must be determined by the uncertainty in…
Probability estimation models play an important role in various fields, such as weather forecasting, recommendation systems, and sports analysis. Among several models estimating probabilities, it is difficult to evaluate which model gives…
Where machine-learned predictive risk scores inform high-stakes decisions, such as bail and sentencing in criminal justice, fairness has been a serious concern. Recent work has characterized the disparate impact that such risk scores can…
Multivariate probabilistic time series forecasts are commonly evaluated via proper scoring rules, i.e., functions that are minimal in expectation for the ground-truth distribution. However, this property is not sufficient to guarantee good…