Related papers: Fisher, Neyman-Pearson or NHST? A Tutorial for Tea…
We introduce Fisher consistency in the sense of unbiasedness as a desirable property for estimators of class prior probabilities. Lack of Fisher consistency could be used as a criterion to dismiss estimators that are unlikely to deliver…
The flexibility and wide applicability of the Fisher randomization test (FRT) makes it an attractive tool for assessment of causal effects of interventions from modern-day randomized experiments that are increasing in size and complexity.…
Labelling data is a major practical bottleneck in training and testing classifiers. Given a collection of unlabelled data points, we address how to select which subset to label to best estimate test metrics such as accuracy, $F_1$ score or…
An active hypothesis testing problem is formulated. In this problem, the agent can perform a fixed number of experiments and then decide on one of the hypotheses. The agent is also allowed to declare its experiments inconclusive if needed.…
This is a writeup, with some elaboration, of the talks by the two authors (a physicist and a statistician) at the first PHYSTAT Informal review on January 24, 2024. We discuss Bayesian and frequentist approaches to dealing with nuisance…
In several interesting applications one is faced with the problem of simultaneous binary hypothesis testing and parameter estimation. Although such joint problems are not infrequent, there exist no systematic analysis in the literature that…
Hyperparameter selection is a critical step in the deployment of artificial intelligence (AI) models, particularly in the current era of foundational, pre-trained, models. By framing hyperparameter selection as a multiple hypothesis testing…
We present the fundamental ideas underlying statistical hypothesis testing using the frequentist framework. We begin with a simple example that builds up the one-sample t-test from the beginning, explaining important concepts such as the…
Semi-supervised learning (SSL) is a popular setting aiming to effectively utilize unlabelled data to improve model performance in downstream natural language processing (NLP) tasks. Currently, there are two popular approaches to make use of…
When statisticians quarrel about hypothesis testing, the debate usually focus on which method is the correct one. The fundamental question of whether we should test hypothesis at all tends to be forgotten. This lack of debate has its roots…
Fisher's likelihood is widely used for statistical inference for fixed unknowns. This paper aims to extend two important likelihood-based methods, namely the maximum likelihood procedure for point estimation and the confidence procedure for…
Imitation learning has enabled robots to perform complex, long-horizon tasks in challenging dexterous manipulation settings. As new methods are developed, they must be rigorously evaluated and compared against corresponding baselines…
This paper raises concerns about the advantages of using statistical significance tests in research assessments as has recently been suggested in the debate about proper normalization procedures for citation indicators. Statistical…
Hypothesis testing plays a central role in statistical inference, and is used in many settings where privacy concerns are paramount. This work answers a basic question about privately testing simple hypotheses: given two distributions $P$…
Matching is a commonly used causal inference study design in observational studies. Through matching on measured confounders between different treatment groups, valid randomization inferences can be conducted under the no unmeasured…
Friedman test is a nonparametric method that proposed for analyzing data from a randomized complete block design as a robust alternative to parametric method and widely applied in many fields such as agriculture, biology, business,…
After some general remarks about the interrelation between philosophical and statistical thinking, the discussion centres largely on significance tests. These are defined as the calculation of $p$-values rather than as formal procedures for…
Fisher's fiducial argument is widely viewed as a failed version of Neyman's theory of confidence limits. But Fisher's goal -- Bayesian-like probabilistic uncertainty quantification without priors -- was more ambitious than Neyman's, and…
We develop a novel computationally efficient and general framework for robust hypothesis testing. The new framework features a new way to construct uncertainty sets under the null and the alternative distributions, which are sets centered…
Statistical hypothesis testing serves as statistical evidence for scientific innovation. However, if the reported results are intentionally biased, hypothesis testing no longer controls the rate of false discovery. In particular, we study…