How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
Abstract
Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since heuristics and hyperparameters substantially vary across different methods, a fair comparison between them can only be achieved by systematically analyzing the influence of these factors. In this paper, we therefore provide a systematic evaluation of the heuristics and hyperparameters that are frequently employed by weight-sharing NAS algorithms. Our analysis uncovers that some commonly-used heuristics for super-net training negatively impact the correlation between super-net and stand-alone performance, and evidences the strong influence of certain hyperparameters and architectural choices. Our code and experiments set a strong and reproducible baseline that future works can build on.
Keywords
Cite
@article{arxiv.2003.04276,
title = {How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS},
author = {Kaicheng Yu and Rene Ranftl and Mathieu Salzmann},
journal= {arXiv preprint arXiv:2003.04276},
year = {2020}
}
Comments
Updated with latest results on NASBench-101, now we achieve 0.48 sparse Kendall-Tau on this space