The success of neural architecture search (NAS) has historically been limited by excessive compute requirements. While modern weight-sharing NAS methods such as DARTS are able to finish the search in single-digit GPU days, extracting the final best architecture from the shared weights is notoriously unreliable. Training-Speed-Estimate (TSE), a recently developed generalization estimator with a Bayesian marginal likelihood interpretation, has previously been used in place of the validation loss for gradient-based optimization in DARTS. This prevents the DARTS skip connection collapse, which significantly improves performance on NASBench-201 and the original DARTS search space. We extend those results by applying various DARTS diagnostics and show several unusual behaviors arising from not using a validation set. Furthermore, our experiments yield concrete examples of the depth gap and topology selection in DARTS having a strongly negative impact on the search performance despite generally receiving limited attention in the literature compared to the operations selection.
@article{arxiv.2112.13023,
title = {DARTS without a Validation Set: Optimizing the Marginal Likelihood},
author = {Miroslav Fil and Binxin Ru and Clare Lyle and Yarin Gal},
journal= {arXiv preprint arXiv:2112.13023},
year = {2021}
}
Comments
Presented at the 5th Workshop on Meta-Learning at NeurIPS 2021