Detecting Where Effects Occur by Testing Hypotheses in Order
Abstract
Experimental evaluations of public policies often randomize a new intervention within many sites or blocks. After a report of an overall result -- statistically significant or not -- the natural question from a policy maker is: \emph{where} did any effects occur? Standard adjustments for multiple testing provide little power to answer this question. In simulations modeled after a 44-block education trial, the Hommel adjustment -- among the most powerful procedures controlling the family-wise error rate (FWER) -- detects effects in only 11\% of truly non-null blocks. We develop a procedure that tests hypotheses top-down through a tree: test the overall null at the root, then groups of blocks, then individual blocks, stopping any branch where the null is not rejected. In the same 44-block design, this approach detects effects in 44\% of non-null blocks -- roughly four times the detection rate. A stopping rule and valid tests at each node suffice for weak FWER control. We show that the strong-sense FWER depends on how rejection probabilities accumulate along paths through the tree. This yields a diagnostic: when power decays fast enough relative to branching, no adjustment is needed; otherwise, an adaptive -adjustment restores control. We apply the method to 25 MDRC education trials and provide an R package, \texttt{manytestsr}.
Cite
@article{arxiv.2602.21068,
title = {Detecting Where Effects Occur by Testing Hypotheses in Order},
author = {Jake Bowers and David Kim and Nuole Chen},
journal= {arXiv preprint arXiv:2602.21068},
year = {2026}
}