Related papers: Why do some probabilistic forecasts lack reliabili…
Conformal Prediction methods have finite-sample distribution-free marginal coverage guarantees. However, they generally do not offer conditional coverage guarantees, which can be important for high-stakes decisions. In this paper, we…
Proper scoring rules are used to assess the out-of-sample accuracy of probabilistic forecasts, with different scoring rules rewarding distinct aspects of forecast performance. Herein, we re-investigate the practice of using proper scoring…
We study the interpretability of conditional probability estimates for binary classification under the agnostic setting or scenario. Under the agnostic setting, conditional probability estimates do not necessarily reflect the true…
The gaussian spread regression model for the calibration of site specific ensemble temperature forecasts depends on the apparently restrictive assumption that the uncertainty around temperature forecasts is normally distributed. We…
This paper demonstrates a methodology for examining the accuracy of uncertain inference systems (UIS), after their parameters have been optimized, and does so for several common UIS's. This methodology may be used to test the accuracy when…
Seasonal climate forecasts are being used increasingly across a range of application sectors. A recent UK governmental report asked: How good are seasonal climate forecasts on a scale of 1-5 (where 5 is very good), and how good can we…
In this paper, we address the probabilistic error quantification of a general class of prediction methods. We consider a given prediction model and show how to obtain, through a sample-based approach, a probabilistic upper bound on the…
Prediction of climate tipping is challenging due to the lack of recent observation of actual climate tipping. Despite many previous efforts to accurately predict the existence and timing of climate tippings under specific climate scenarios,…
Probability models are only useful at explaining the uncertainty of what we do not know, and should never be used to say what we already know. Probability and statistical models are useless at discerning cause. Classical statistical…
Solar flares occur in complex sunspot groups, but it remains unclear how the probability of producing a flare of a given magnitude relates to the characteristics of the sunspot group. Here, we use Geostationary Operational Environmental…
Conformal prediction provides model-agnostic and distribution-free uncertainty quantification through prediction sets that are guaranteed to include the ground truth with any user-specified probability. Yet, conformal prediction is not…
Attempts to replicate probabilistic reasoning in expert systems have typically overlooked a critical ingredient of that process. Probabilistic analysis typically requires extensive judgments regarding interdependencies among hypotheses and…
Probability forecasts of events are routinely used in climate predictions, in forecasting default probabilities on bank loans or in estimating the probability of a patient's positive response to treatment. Scoring rules have long been used…
In various applications, probabilistic forecasts are required to quantify the inherent uncertainty associated with the forecast. However, numerous modern forecasting methods are still designed to create deterministic forecasts. Transforming…
The evaluation of probabilistic forecasts plays a central role both in the interpretation and in the use of forecast systems and their development. Probabilistic scores (scoring rules) provide statistical measures to assess the quality of…
The field of distribution-free predictive inference provides tools for provably valid prediction without any assumptions on the distribution of the data, which can be paired with any regression algorithm to provide accurate and reliable…
The intrinsic difficulties in building realistic climate models and in providing complete, reliable and meaningful observational datasets, and the conceptual impossibility of testing theories against data imply that the usual Galilean…
When providing probabilistic forecasts for uncertain future events, it is common to strive for calibrated forecasts, that is, the predictive distribution should be compatible with the observed outcomes. Several notions of calibration are…
Current techniques for predicting climate change are mainly based on "massive" deterministic numerical modeling. However, the ocean-atmosphere system is a so-called "complex system", made up of a large number of interacting elements. We…
Consistency, in a narrow sense, denotes the alignment between the forecast-optimization strategy and the verification directive. The current recommended deterministic solar forecast verification practice is to report the skill score based…