Statistical isotropy of the universe and the look-elsewhere effect

Recently, Jones et al. [arXiv:2310.12859] claimed strong evidence for the statistical anisotropy of the universe. The claim is based on a joint analysis of four different anomaly tests of the cosmic microwave background data, each of which is known to be anomalous, with a lower level of significance. They reported a combined $p$-value of about $3\times 10^{-8}$, which is more than a $5\sigma$ level of significance. We observe that statistical anisotropy is not even relevant for two of the four considered tests, which seems sufficient to invalidate the authors' claim. Furthermore, even if one reinterprets the claim as evidence against $\Lambda$CDM rather than statistical anisotropy, we argue that this result significantly suffers from the look-elsewhere effect. Assuming a set of independent (i.e., uncorrelated) tests, we show that if the four tests with the smallest $p$-values are cherry-picked from 10 independent tests, the $p$-value reported by Jones et al. corresponds to only $3\sigma$ significance. If there are 27 independent tests, the significance falls to $2\sigma$. These numbers, however, overstate our argument, since the four tests used by Jones et al. are slightly correlated. Determining the correlation of Jones et al.'s tests by comparing their joint $p$-value with the product of the four separate $p$-values, we find that about 16 or 50 tests are sufficient to reduce the significance of Jones et al.'s results to 3$\sigma$ or 2$\sigma$ significance, respectively. We also provide a list of anomaly tests discussed in the literature (and propose a few generalizations), suggesting that very plausibly 16 (or even 50) independent tests have been published, and possibly many more have been considered but not published. We conclude that the current data is consistent with the $\Lambda$CDM model and, in particular, with statistical isotropy.