Related papers: Thou Shalt Not Reject the P-value

Massive numbers of meta-analysis studies are being published. A Google Scholar search of "systematic review and meta-analysis" returns about 452k hits since 2014. The search was done on Jan 14, 2019. There is a need to have some way to…

With the adoption of machine learning-based solutions in routine clinical practice, the need for reliable interpretability tools has become pressing. Shapley values provide local explanations. The method gained popularity in recent years.…

This paper addresses the statistical significance of structures in random data: Given a set of vectors and a measure of mutual similarity, how likely does a subset of these vectors form a cluster with enhanced similarity among its elements?…

We provide accessible insight into the current 'replication crisis' in 'statistical science', by revisiting the old metaphor of 'court trial as hypothesis test'. Inter alia, we define and diagnose harmful statistical witch-hunting both in…

We seek to conduct statistical inference for a large collection of primary parameters, each with its own nuisance parameters. Our approach is partially Bayesian, in that we treat the primary parameters as fixed while we model the nuisance…

Hypothesis testing is a central statistical method in psychological research and the cognitive sciences. While the problems of null hypothesis significance testing (NHST) have been debated widely, few attractive alternatives exist. In this…

This article gives a conceptual review of the e-value, ev(H|X) -- the epistemic value of hypothesis H given observations X. This statistical significance measure was developed in order to allow logically coherent and consistent tests of…

Particle physics experiments such as those run in the Large Hadron Collider result in huge quantities of data, which are boiled down to a few numbers from which it is hoped that a signal will be detected. We discuss a simple probability…

The influential claim that most published results are false raised concerns about the trustworthiness and integrity of science. Since then, there have been numerous attempts to examine the rate of false-positive results that have failed to…

Bayesian variable selection is a powerful tool for data analysis, as it offers a principled method for variable selection that accounts for prior information and uncertainty. However, wider adoption of Bayesian variable selection has been…

This paper introduces an open-ended sequential algorithm for computing the p-value of a test using Monte Carlo simulation. It guarantees that the resampling risk, the probability of a different decision than the one based on the theoretical…

The randomized $p$-value, (nonrandomized) mid-$p$-value and abstract randomized $p$-value have all been recommended for testing a null hypothesis whenever the test statistic has a discrete distribution. This paper provides a unifying…

A lot of Machine Learning (ML) and Deep Learning (DL) research is of an empirical nature. Nevertheless, statistical significance testing (SST) is still not widely used. This endangers true progress, as seeming improvements over a baseline…

Null hypothesis significance testing remains popular despite decades of concern about misuse and misinterpretation. We believe that much of the problem is due to language: significance testing has little to do with other meanings of the…

The energy test method is a multi-dimensional test of whether two samples are consistent with arising from the same underlying population, through the calculation of a single test statistic (called the $T$-value). The method has recently…

The problem of combining p-values is an old and fundamental one, and the classic assumption of independence is often violated or unverifiable in many applications. There are many well-known rules that can combine a set of arbitrarily…

Conformal prediction is a powerful framework for distribution-free uncertainty quantification. The standard approach to conformal prediction relies on comparing the ranks of prediction scores: under exchangeability, the rank of a future…

Hypothesis tests calibrated by (re)sampling methods (such as permutation, rank and bootstrap tests) are useful tools for statistical analysis, at the computational cost of requiring Monte-Carlo sampling for calibration. It is common and…

Pursuing a scientific idea is often justified by the promise associated with it. Philosophers of science have proposed a variety of approaches to such promise, including more specific indicators. Economic models in particular emphasise the…

The high fraction of published results that turn out to be incorrect is a major concern of today's science. This paper contributes to the understanding of this problem in two independent directions. First, Johnson's recent claim that…