Related papers: On admissibility in post-hoc hypothesis testing
Selective inference is a subfield of statistics that enables valid inference after selection of a data-dependent question. In this paper, we introduce selectively dominant p-values, a class of p-values that allow practitioners to easily…
Adaptive experiments use preliminary analyses of the data to inform further course of action and are commonly used in many disciplines including medical and social sciences. Because the null hypothesis and experimental design are…
A common problem in machine learning is determining if a variable significantly contributes to a model's prediction performance. This problem is aggravated for datasets, such as gene expression datasets, that suffer the worst case of…
A/B testing is ubiquitous within the machine learning and data science operations of internet companies. Generically, the idea is to perform a statistical test of the hypothesis that a new feature is better than the existing platform---for…
Permutation tests are widely used for statistical hypothesis testing when the sampling distribution of the test statistic under the null hypothesis is analytically intractable or unreliable due to finite sample sizes. One critical challenge…
Feature importance scores are ubiquitous tools for understanding the predictions of machine learning models. However, many popular attribution methods suffer from high instability due to random sampling. Leveraging novel ideas from…
The current definition of a Bayesian credible set cannot, in general, achieve an arbitrarily preassigned credible level. This drawback is particularly acute for classification problems, where there are only a finite number of achievable…
Importance sampling approximates expectations with respect to a target measure by using samples from a proposal measure. The performance of the method over large classes of test functions depends heavily on the closeness between both…
We put forward an adaptive alpha (Type I Error) that decreases as the information grows, for hypothesis tests in which nested linear models are compared. A less elaborate adaptation was already presented in \citet{PP2014} for comparing…
Statistical significance measures the reliability of a result obtained from a random experiment. We investigate the number of repetitions needed for a statistical result to have a certain significance. In the first step, we consider…
Most of the literature on change-point analysis by means of hypothesis testing considers hypotheses of the form H0 : \theta_1 = \theta_2 vs. H1 : \theta_1 != \theta_2, where \theta_1 and \theta_2 denote parameters of the process before and…
Statistics has made tremendous advances since the times of Fisher, Neyman, Jeffreys, and others, but the fundamental and practically relevant questions about probability and inference that puzzled our founding fathers remain unanswered. To…
Hypothesis tests are a crucial statistical tool for data mining and are the workhorse of scientific research in many fields. Here we study differentially private tests of independence between a categorical and a continuous variable. We take…
A number of biomedical problems require performing many hypothesis tests, with an attendant need to apply stringent thresholds. Often the data take the form of a series of predictor vectors, each of which must be compared with a single…
In the last months, due to the emergency of Covid-19, questions related to the fact of belonging or not to a particular class of individuals (`infected or not infected'), after being tagged as `positive' or `negative' by a test, have never…
We describe a general framework for weighted parametric multiple test procedures based on the closure principle. We utilize general weighting strategies that can reflect complex study objectives and include many procedures in the literature…
We consider a problem of simple hypothesis testing using a randomized test via a tunable loss function proposed by Liao \textit{et al}. In this problem, we derive results that correspond to the Neyman--Pearson lemma, the Chernoff--Stein…
Quantifying uncertainty in detected changepoints is an important problem. However it is challenging as the naive approach would use the data twice, first to detect the changes, and then to test them. This will bias the test, and can lead to…
There exist a number of tests for assessing the nonparametric heteroscedastic location-scale assumption. Here we consider a goodness-of-fit test for the more general hypothesis of the validity of this model under a parametric functional…
We introduce probability estimation, a broadly applicable framework to certify randomness in a finite sequence of measurement results without assuming that these results are independent and identically distributed. Probability estimation…