Albeit being a prevalent architecture searching approach, differentiable architecture search (DARTS) is largely hindered by its substantial memory cost since the entire supernet resides in the memory. This is where the single-path DARTS comes in, which only chooses a single-path submodel at each step. While being memory-friendly, it also comes with low computational costs. Nonetheless, we discover a critical issue of single-path DARTS that has not been primarily noticed. Namely, it also suffers from severe performance collapse since too many parameter-free operations like skip connections are derived, just like DARTS does. In this paper, we propose a new algorithm called RObustifying Memory-Efficient NAS (ROME) to give a cure. First, we disentangle the topology search from the operation search to make searching and evaluation consistent. We then adopt Gumbel-Top2 reparameterization and gradient accumulation to robustify the unwieldy bi-level optimization. We verify ROME extensively across 15 benchmarks to demonstrate its effectiveness and robustness.
@article{arxiv.2011.11233,
title = {ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulation},
author = {Xiaoxing Wang and Xiangxiang Chu and Yuda Fan and Zhexi Zhang and Bo Zhang and Xiaokang Yang and Junchi Yan},
journal= {arXiv preprint arXiv:2011.11233},
year = {2023}
}