Related papers: On admissibility in post-hoc hypothesis testing
Testing by betting has been a cornerstone of the game-theoretic statistics literature. One bets against the null hypothesis, and the accumulated wealth $W_t$ quantifies the evidence against the null hypothesis after $t$ rounds, and the null…
Statistically equivalent blocks are not frequently considered in the context of nonparametric two-sample hypothesis testing. Despite the limited exposure, this paper shows that a number of classical nonparametric hypothesis tests can be…
Equivalence testing is of emerging importance in genomics studies but has hitherto been little studied in this content. In this paper, we define the notion of equivalence of gene expression and determine a `strength of evidence' measure for…
While generative models, especially large language models (LLMs), are ubiquitous in today's world, principled mechanisms to assess their (in)correctness are limited. Using the conformal prediction framework, previous works construct sets of…
Bayesian hypothesis testing is re-examined from the perspective of an a priori assessment of the test statistic distribution under the alternative. By assessing the distribution of an observable test statistic, rather than prior parameter…
Hypothesis testing plays a central role in statistical inference, and is used in many settings where privacy concerns are paramount. This work answers a basic question about privately testing simple hypotheses: given two distributions $P$…
Post-hoc interpretability methods are critical tools to explain neural-network results. Several post-hoc methods have emerged in recent years, but when applied to a given task, they produce different results, raising the question of which…
A hypothesis testing algorithm is replicable if, when run on two different samples from the same distribution, it produces the same output with high probability. This notion, defined by by Impagliazzo, Lei, Pitassi, and Sorell [STOC'22],…
In contrast to its common definition and calculation, interpretation of p-values diverges among statisticians. Since p-value is the basis of various methodologies, this divergence has led to a variety of test methodologies and evaluations…
A unified framework is proposed for tests of unobserved heterogeneity in parametric statistic models based on Neyman's $C(\alpha)$ approach. Such tests are irregular in the sense that the first order derivative of the log likelihood with…
This paper studies the problem of testing if an input (Gamma,*), where Gamma is a finite set of unknown size and * is a binary operation over Gamma given as an oracle, is close to a specified class of groups. Friedl et al. [Efficient…
Generative surveying -- where collections of LLM-based personas provide feedback on messages -- has emerged as a cheap and scalable alternative to traditional market research. However, LLMs are sensitive to small variations in prompt design…
While statistics and machine learning offers numerous methods for ensuring generalization, these methods often fail in the presence of adaptivity---the common practice in which the choice of analysis depends on previous interactions with…
As increasingly complex hypothesis-testing scenarios are considered in many scientific fields, analytic derivation of null distributions is often out of reach. To the rescue comes Monte Carlo testing, which may appear deceptively simple: as…
We present a semantics for adding uncertainty to conditional logics for default reasoning and belief revision. We are able to treat conditional sentences as statements of conditional probability, and express rules for revision such as "If A…
This paper is a journal counterpart to our FOCS 2021 paper, in which we initiate the study of property testing problems concerning a finite system of relations $E$ between permutations, generalizing the study of stability in permutations.…
We study the properties of several likelihood-based statistics commonly used in testing for the presence of a known signal under a mixture model with known background, but unknown signal fraction. Under the null hypothesis of no signal, all…
In contemporary problems involving genetic or neuroimaging data, thousands of hypotheses need to be tested. Due to their high power, and finite sample guarantees on type-I error under weak assumptions, Monte Carlo permutation tests are…
In order to determine whether or not an effect is absent based on a statistical test, the recommended frequentist tool is the equivalence test. Typically, it is expected that an appropriate equivalence margin has been specified before any…
In many applications, it is desirable that a classifier not only makes accurate predictions, but also outputs calibrated posterior probabilities. However, many existing classifiers, especially deep neural network classifiers, tend to be…