Related papers: Backtesting forecast accuracy
Explicitly accounting for all applicable independent variables, even when the model being tested does not, is critical in testing goodness-of-fit for logistic regression. This can increase statistical power by orders of magnitude.
The impact of including postcursors in the null hypothesis test is discussed. Unequal prediction probabilities can be included in the null hypothesis test using a generalization of the central limit theorem. A test for determining the…
We propose randomization tests of whether forecast 1 outperforms forecast 2 across a class of scoring functions. This hypothesis is of applied interest: While the prediction context often prescribes a certain class of scoring functions, it…
When does a machine learning model predict the future of individuals and when does it recite patterns that predate the individuals? In this work, we propose a distinction between these two pathways of prediction, supported by theoretical,…
Accurately predicting faulty software units helps practitioners target faulty units and prioritize their efforts to maintain software quality. Prior studies use machine-learning models to detect faulty software code. We revisit past studies…
Risk evaluation is a forecast, and its validity must be backtested. Probability distribution forecasts are used in this work and allow for more powerful validations compared to point forecasts. Our aim is to use bivariate copulas in order…
Robustness checks are routine in empirical work, but there is no standard statistical procedure to formally measure what one can learn from them. I propose a "robustness radius" measure to quantify the amount by which the robustness checks…
Statistical methods are based on model assumptions, and it is statistical folklore that a method's model assumptions should be checked before applying it. This can be formally done by running one or more misspecification tests of model…
The choice of model class is fundamental in statistical learning and system identification, no matter whether the class is derived from physical principles or is a generic black-box. We develop a method to evaluate the specified model class…
In this paper we address the statistical problem of testing if a stationary process is Gaussian. The observation consists in a finite sample path of the process. Using a random projection technique introduced and studied in Cuesta-Albertos…
Assessment of replicability is critical to ensure the quality and rigor of scientific research. In this paper, we discuss inference and modeling principles for replicability assessment. Targeting distinct application scenarios, we propose…
Test-negative designs are widely used for post-market evaluation of vaccine effectiveness, particularly in cases when randomized trials are not feasible. Differing from classical test-negative designs where only healthcare-seekers with…
The status of precision tests of the Standard Model is reviewed.
Machine learning models are central to people's lives and impact society in ways as fundamental as determining how people access information. The gravity of these models imparts a responsibility to model developers to ensure that they are…
Bayesian hypothesis testing is investigated when the prior probabilities of the hypotheses, taken as a random vector, are quantized. Nearest neighbor and centroid conditions are derived using mean Bayes risk error as a distortion measure…
A common problem in numerous research areas, particularly in clinical trials, is to test whether the effect of an explanatory variable on an outcome variable is equivalent across different groups. In practice, these tests are frequently…
In this paper, we investigate the hypothesis testing problem that checks whether part of covariates / confounders significantly affect the heterogeneous treatment effect given all covariates. This model checking is particularly useful in…
Nested error regression models are useful tools for analysis of grouped data, especially in the case of small area estimation. This paper suggests a nested error regression model using uncertain random effects in which the random effect in…
We consider the problem of evaluating the performance of a decision policy using past observational data. The outcome of a policy is measured in terms of a loss (aka. disutility or negative reward) and the main problem is making valid…
While the {estimation} of risk is an important question in the daily business of banking and insurance, many existing plug-in estimation procedures suffer from an unnecessary bias. This often leads to the underestimation of risk and…