The limited and dynamically varied resources on edge devices motivate us to deploy an optimized deep neural network that can adapt its sub-networks to fit in different resource constraints. However, existing works often build sub-networks through searching different network architectures in a hand-crafted sampling space, which not only can result in a subpar performance but also may cause on-device re-configuration overhead. In this paper, we propose a novel training algorithm, Dynamic REal-time Sparse Subnets (DRESS). DRESS samples multiple sub-networks from the same backbone network through row-based unstructured sparsity, and jointly trains these sub-networks in parallel with weighted loss. DRESS also exploits strategies including parameter reusing and row-based fine-grained sampling for efficient storage consumption and efficient on-device adaptation. Extensive experiments on public vision datasets show that DRESS yields significantly higher accuracy than state-of-the-art sub-networks.
@article{arxiv.2207.00670,
title = {DRESS: Dynamic REal-time Sparse Subnets},
author = {Zhongnan Qu and Syed Shakib Sarwar and Xin Dong and Yuecheng Li and Ekin Sumbul and Barbara De Salvo},
journal= {arXiv preprint arXiv:2207.00670},
year = {2022}
}
Comments
Published in Efficient Deep Learning for Computer Vision (ECV) CVPR Workshop 2022