Related papers: Backtesting forecast accuracy
This paper is concerned with a Bayesian approach to testing hypotheses in statistical inverse problems. Based on the posterior distribution $\Pi \left(\cdot |Y = y\right)$, we want to infer whether a feature $\langle\varphi,…
We introduce a new approach for comparing the predictive accuracy of two nested models that bypasses the difficulties caused by the degeneracy of the asymptotic variance of forecast error loss differentials used in the construction of…
No matter the nature of the response and/or explanatory variables in a regression model, some basic issues such as the existence of an effect of the predictor on the response, or the assessment of a common shape across groups of…
Fitting models to data is an important part of the practice of science. Advances in machine learning have made it possible to fit more -- and more complex -- models, but have also exacerbated a problem: when multiple models fit the data…
A framework is developed using techniques from rate distortion theory in statistical testing. The idea is first to do optimal compression according to a certain distortion function and then use information divergence from the compressed…
A new test for measuring the accuracy of financial market risk estimations is introduced. It is based on the probability integral transform (PIT) of the ex post realized returns using the ex ante probability distributions underlying the…
Two procedures for checking Bayesian models are compared using a simple test problem based on the local Hubble expansion. Over four orders of magnitude, p-values derived from a global goodness-of-fit criterion for posterior probability…
Statistical power is a measure of the replicability of a categorical hypothesis test. Formally, it is the probability of detecting an effect, if there is a true effect present in the population. Hence, optimizing statistical power as a…
While running any experiment, we often have to consider the statistical power to ensure an effective study. Statistical power or power ensures that we can observe an effect with high probability if such a true effect exists. However,…
Regression is one of the most commonly used statistical techniques. However, testing regression systems is a great challenge because of the absence of test oracle in general. In this paper, we show that Metamorphic Testing is an effective…
We study robust estimators of the mean of a probability measure $P$, called robust empirical mean estimators. This elementary construction is then used to revisit a problem of aggregation and a problem of estimator selection, extending…
The workshop is devoted to model-based testing of both software and hardware. Model-based testing uses models describing the required behavior of the system under consideration to guide such efforts as test selection and test results…
In a regression model, prediction is typically performed after model selection. The large variability in the model selection makes the prediction unstable. Thus, it is essential to reduce the variability in model selection and improve…
The measurement of the efficiency of an event selection is always an important part of the analysis of experimental data. The statistical techniques which are needed to determine the efficiency and its uncertainty are reviewed. Frequentist…
Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common practice of basing inferences solely on p-values has been under intense criticism for over…
Goodness-of-fit tests are often used in data analysis to test the agreement of a distribution to a set of data. These tests can be used to detect an unknown signal against a known background or to set limits on a proposed signal…
A long noted difficulty when assessing the reliability (or calibration) of forecasting systems is that reliability, in general, is a hypothesis not about a finite dimensional parameter but about an entire functional relationship. A…
In a well-calibrated risk prediction model, the average predicted probability is close to the true event rate for any given subgroup. Such models are reliable across heterogeneous populations and satisfy strong notions of algorithmic…
A Bayes factor is proposed for testing whether the effect of a key predictor variable on the dependent variable is linear or nonlinear, possibly while controlling for certain covariates. The test can be used (i) when one is interested in…
Surrogate data testing is a method frequently applied to evaluate the results of nonlinear time series analysis. Since the null hypothesis tested against is a linear, gaussian, stationary stochastic process a positive outcome may not only…