Weight-sharing (WS) has recently emerged as a paradigm to accelerate the automated search for efficient neural architectures, a process dubbed Neural Architecture Search (NAS). Although very appealing, this framework is not without drawbacks and several works have started to question its capabilities on small hand-crafted benchmarks. In this paper, we take advantage of the \nasbench dataset to challenge the efficiency of WS on a representative search space. By comparing a SOTA WS approach to a plain random search we show that, despite decent correlations between evaluations using weight-sharing and standalone ones, WS is only rarely significantly helpful to NAS. In particular we highlight the impact of the search space itself on the benefits.
@article{arxiv.2002.04289,
title = {To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing},
author = {Aloïs Pourchot and Alexis Ducarouge and Olivier Sigaud},
journal= {arXiv preprint arXiv:2002.04289},
year = {2020}
}