It is commonly accepted that the Vision Transformer model requires sophisticated regularization techniques to excel at ImageNet-1k scale data. Surprisingly, we find this is not the case and standard data augmentation is sufficient. This note presents a few minor modifications to the original Vision Transformer (ViT) vanilla training setting that dramatically improve the performance of plain ViT models. Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, and 300 epochs of training reach 80% in less than one day.
@article{arxiv.2205.01580,
title = {Better plain ViT baselines for ImageNet-1k},
author = {Lucas Beyer and Xiaohua Zhai and Alexander Kolesnikov},
journal= {arXiv preprint arXiv:2205.01580},
year = {2022}
}
Comments
Code available at https://github.com/google-research/big_vision