Related papers: Potential root mean square error skill score

We initiate the study of the truthfulness of calibration measures in sequential prediction. A calibration measure is said to be truthful if the forecaster (approximately) minimizes the expected penalty by predicting the conditional…

Randomized smoothing has shown promising certified robustness against adversaries in classification tasks. Despite such success with only zeroth-order access to base models, randomized smoothing has not been extended to a general form of…

The estimation of covariance matrices of multiple classes with limited training data is a difficult problem. The sample covariance matrix (SCM) is known to perform poorly when the number of variables is large compared to the available…

We adopt and expand McDonald's (2011) regression framework for measurement precision, integrating two key perspectives: (a) reliability of observed scores and (b) optimal prediction of latent scores. Reliability arises from a measurement…

When climate forecasts are highly uncertain, the optimal mean squared error strategy is to ignore them. When climate forecasts are highly certain, the optimal mean squared error strategy is to use them as is. In between these two extremes…

In this paper we propose a novel approach to realize forecast verification. Specifically, we introduce a strategy for assessing the severity of forecast errors based on the evidence that, on the one hand, a false alarm just anticipating an…

The use of solar photovoltaics (PV) energy provides additional resources to the electric power grid. The downside of this integration is that the solar power supply is unreliable and highly dependent on the weather condition. The…

To use machine learning in high stakes applications (e.g. medicine), we need tools for building confidence in the system and evaluating whether it is reliable. Methods to improve model reliability often require new learning algorithms (e.g.…

In machine learning forecasting, standard error metrics such as mean absolute error (MAE) and mean squared error (MSE) quantify discrepancies between predictions and target values. However, these metrics do not directly evaluate the…

For finite samples with binary outcomes penalized logistic regression such as ridge logistic regression (RR) has the potential of achieving smaller mean squared errors (MSE) of coefficients and predictions than maximum likelihood…

In situations where forecasters are scored on the quality of their probabilistic predictions, it is standard to use `proper' scoring rules to perform such scoring. These rules are desirable because they give forecasters no incentive to lie…

In deploying artificial intelligence (AI) models, selective prediction offers the option to abstain from making a prediction when uncertain about model quality. To fulfill its promise, it is crucial to enforce strict and precise error…

Randomized quadratures for integrating functions in Sobolev spaces of order $\alpha \ge 1$, where the integrability condition is with respect to the Gaussian measure, are considered. In this function space, the optimal rate for the…

Updating machine learning models with new information usually improves their predictive performance, yet, in many applications, it is also desirable to avoid changing the model predictions too much. This property is called stability. In…

Accurate solar energy output prediction is key for integrating renewables into grids, maintaining stability, and improving energy management. However, standard error metrics such as Root Mean Squared Error (RMSE), Mean Absolute Error (MAE),…

Previous analysis on forecasting theory either assume knowing the true parameters or assume the stationarity of the series. Not much are known on the forecasting theory for nonstationary process with estimated parameters. This paper…

The recent developments in the machine learning domain have enabled the development of complex multivariate probabilistic forecasting models. Therefore, it is pivotal to have a precise evaluation method to gauge the performance and…

Maximum likelihood estimation in nonlinear models can exhibit substantial instability in finite samples when the data provide limited information about certain parameters. Such instability is driven by rare but extreme realizations of the…

Smooth power generation from solar stations demand accurate, reliable and efficient forecast of solar energy for optimal integration to cater market demand; however, the implicit instability of solar energy production may cause serious…

A recent technique of randomized smoothing has shown that the worst-case (adversarial) $\ell_2$-robustness can be transformed into the average-case Gaussian-robustness by "smoothing" a classifier, i.e., by considering the averaged…