Pure transformers have shown great potential for vision tasks recently. However, their accuracy in small or medium datasets is not satisfactory. Although some existing methods introduce a CNN as a teacher to guide the training process by distillation, the gap between teacher and student networks would lead to sub-optimal performance. In this work, we propose a new One-shot Vision transformer search framework with Online distillation, namely OVO. OVO samples sub-nets for both teacher and student networks for better distillation results. Benefiting from the online distillation, thousands of subnets in the supernet are well-trained without extra finetuning or retraining. In experiments, OVO-Ti achieves 73.32% top-1 accuracy on ImageNet and 75.2% on CIFAR-100, respectively.
@article{arxiv.2212.13766,
title = {OVO: One-shot Vision Transformer Search with Online distillation},
author = {Zimian Wei and Hengyue Pan and Xin Niu and Dongsheng Li},
journal= {arXiv preprint arXiv:2212.13766},
year = {2023}
}