Considerations in the Interpretation of Cosmological Anomalies

Anomalies drive scientific discovery -- they are associated with the cutting edge of the research frontier, and thus typically exploit data in the low signal-to-noise regime. In astronomy, the prevalence of systematics --- both "known unknowns" and "unknown unknowns" --- combined with increasingly large datasets, the widespread use of ad hoc estimators for anomaly detection, and the "look-elsewhere" effect, can lead to spurious false detections. In this informal note, I argue that anomaly detection leading to discoveries of new physics requires a combination of physical understanding, careful experimental design to avoid confirmation bias, and self-consistent statistical methods. These points are illustrated with several concrete examples from cosmology.