Related papers: Small-sample corrections for score tests in Birnba…
We review the alternative proposals introduced recently in the literature to update the standard formula to estimate the uncertainty on the mean of repeated measurements, and we compare their performances on synthetic examples with normal…
We describe how to calculate standard errors for A/B tests that include clustered data, ratio metrics, and/or covariate adjustment. We may do this for power analysis/sample size calculations prior to running an experiment using historical…
Plausibility is a formalization of exact tests for parametric models and generalizes procedures such as Fisher's exact test. The resulting tests are based on cumulative probabilities of the probability density function and evaluate…
Nonprobability (convenience) samples are increasingly sought to stabilize estimations for one or more population variables of interest that are performed using a randomized survey (reference) sample by increasing the effective sample size.…
I present a frequentist method for quantifying uncertainty when correcting correlations for attenuation due to measurement error. The method is conservative but has far better coverage properties than the methods currently used when sample…
In modern scientific research, small-scale studies with limited participants are increasingly common. However, interpreting individual outcomes can be challenging, making it standard practice to combine data across studies using random…
Items in a test are often used as a basis for making decisions and such tests are therefore required to have good psychometric properties, like unidimensionality. In many cases the sum score is used in combination with a threshold to decide…
Risk evaluation is a forecast, and its validity must be backtested. Probability distribution forecasts are used in this work and allow for more powerful validations compared to point forecasts. Our aim is to use bivariate copulas in order…
We study ratio metrics in A/B testing at the presence of correlation among observations coming from the same user and provides practical guidance especially when two metrics contradict each other. We propose new estimating methods to…
Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked…
Although the log-likelihood is widely used in model selection, the log-likelihood ratio has had few applications in this area. We develop a log-likelihood ratio based method for selecting regression models by focusing on the set of models…
The objective of this work is to quantify the uncertainty in probability of failure estimates resulting from incomplete knowledge of the probability distributions for the input random variables. We propose a framework that couples the…
We present a new, analytic, Poisson likelihood derived, technique to account for the statistical uncertainties inherent in simulation samples of limited size. This method has better coverage properties than other techniques, is valid for…
Simulation-based calibration checking (SBC) is a practical method to validate computationally-derived posterior distributions or their approximations. In this paper, we introduce a new variant of SBC to alleviate several known problems. Our…
There is a fundamental disconnect between what is tested in a model adequacy test, and what we would like to test. The usual approach is to test the null hypothesis "Model M is the true model." However, Model M is never the true model. A…
Control rate regression is a diffuse approach to account for heterogeneity among studies in meta-analysis by including information about the outcome risk of patients in the control condition. Correcting for the presence of measurement error…
Corrected confidence intervals are developed for the mean of the second component of a bivariate normal process when the first component is being monitored sequentially. This is accomplished by constructing a first approximation to a…
A formal likelihood ratio hypothesis test for the validity of a parametric regression function is proposed, using a large-dimensional, nonparametric double cone alternative. For example, the test against a constant function uses the…
We introduce equivalence testing procedures for linear regression analyses. Such tests can be very useful for confirming the lack of a meaningful association between a continuous outcome and a continuous or binary predictor. Specifically,…
A profile likelihood ratio test is proposed for inferences on the index coefficients in generalized single-index models. Key features include its simplicity in implementation, invariance against parametrization, and exhibiting substantially…