Do GRE scores help predict getting a physics Ph.D.? A comment on a paper by Miller et al
Abstract
A recent paper in Sci. Adv. by Miller et al. concludes that GREs do not help predict whether physics grad students will get Ph.D.s. The paper makes numerous elementary statistics errors, including introduction of unnecessary collider-like stratification bias, variance inflation by collinearity and range restriction, omission of needed data (some subsequently provided), a peculiar choice of null hypothesis on subgroups, blurring the distinction between failure to reject a null and accepting a null, and an extraordinary procedure for radically inflating confidence intervals in a figure. Release of results of simpler models, e.g. without unnecessary stratification, would fix some key problems. The paper exhibits exactly the sort of research techniques which we should be teaching students to avoid.
Cite
@article{arxiv.1902.09442,
title = {Do GRE scores help predict getting a physics Ph.D.? A comment on a paper by Miller et al},
author = {Michael B. Weissman},
journal= {arXiv preprint arXiv:1902.09442},
year = {2020}
}
Comments
24p@1.5_spacing. v2:more refs. v3 clears up units. v4 finds another error in Miller paper. v5:more refs v6: more quantitative re collider bias. v7: another ref. & tweaks range restriction argument. v8: more quantitive on collinearity & GRE predictive power. v9 uses newly released data v10 adds reply to published response. see Sci Adv EAAX3787 (2020)