Related papers: Ranking earthquake forecasts using proper scoring …
Pairwise comparisons are widely used in decision analysis, preference modeling, and evaluation problems. In many practical situations, the observed comparison matrix is not reciprocal. This lack of reciprocity is often treated as a defect…
In this study, we introduced various statistical performance metrics, based on the pinball loss and the empirical coverage, for the ranking of probabilistic forecasting models. We tested the ability of the proposed metrics to determine the…
In recent years, probabilistic forecasting is an emerging topic, which is why there is a growing need of suitable methods for the evaluation of multivariate predictions. We analyze the sensitivity of the most common scoring rules,…
Scoring rules measure the deviation between a probabilistic forecast and reality. Strictly proper scoring rules have the property that for any forecast, the mathematical expectation of the score of a forecast p by the lights of p is…
The paper by Taroni et al. (2016) considers results of forward prediction of Italian strong earthquakes by CN algorithm with the declared intent of providing "a careful assessment of CN prediction performances... using standard testing…
The assessment of seismic risk in urban areas with high seismicity is certainly one of the most important problems that territorial managers have to face. A reliable evaluation of this risk is the basis for the design of both specific…
The classical paradigm of scoring rules is to discriminate between two different forecasts by comparing them with observations. The probability distribution of the observed record is assumed to be perfect as a verification benchmark. In…
We present a simple theoretical framework, and corresponding practical procedures, for comparing probabilistic models on real data in a traditional machine learning setting. This framework is based on the theory of proper scoring rules, but…
Neural Posterior Estimation methods for simulation-based inference can be ill-suited for dealing with posterior distributions obtained by conditioning on multiple observations, as they tend to require a large number of simulator calls to…
Estimates of seismic wave speeds in the Earth (seismic velocity models) are key input parameters to earthquake simulations for ground motion prediction. Owing to the non-uniqueness of the seismic inverse problem, typically many velocity…
We construct a model of expert prediction where predictions can influence the state of the world. Under this model, we show through theoretical and numerical results that proper scoring rules can incentivize experts to manipulate the world…
Competing risks occur in survival analysis when multiple causes of death are present. They play a prominent role in several domains extending beyond biostatistics to encompass epidemiology, actuarial sciences, and reliability theory. This…
Proper scoring rules are commonly applied to quantify the accuracy of distribution forecasts. Given an observation they assign a scalar score to each distribution forecast, with the the lowest expected score attributed to the true…
Immediately following a disaster event, such as an earthquake, estimates of the damage extent play a key role in informing the coordination of response and recovery efforts. We develop a novel impact estimation tool that leverages a…
Though athletics statistics are abundant, it is a difficult task to quantitatively compare performances from different events of track, field, and road running in a meaningful way. There are several commonly-used methods, but each has its…
A typical situation in competing risks analysis is that the researcher is only interested in a subset of risks. This paper considers a depending competing risks model with the distribution of one risk being a parametric or semi-parametric…
Decision-making often involves ranking and selection. For example, to assemble a team of political forecasters, we might begin by narrowing our choice set to the candidates we are confident rank among the top 10% in forecasting ability.…
In the face of uncertainty, the need for probabilistic assessments has long been recognized in the literature on forecasting. In classification, however, comparative evaluation of classifiers often focuses on predictions specifying a single…
Probability forecasts for binary events play a central role in many applications. Their quality is commonly assessed with proper scoring rules, which assign forecasts a numerical score such that a correct forecast achieves a minimal…
The comparison of benchmark error sets is an essential tool for the evaluation of theories in computational chemistry. The standard ranking of methods by their Mean Unsigned Error is unsatisfactory for several reasons linked to the…