English

ViT-5: Vision Transformers for The Mid-2020s

Computer Vision and Pattern Recognition 2026-02-10 v1

Abstract

This work presents a systematic investigation into modernizing Vision Transformer backbones by leveraging architectural advancements from the past five years. While preserving the canonical Attention-FFN structure, we conduct a component-wise refinement involving normalization, activation functions, positional encoding, gating mechanisms, and learnable tokens. These updates form a new generation of Vision Transformers, which we call ViT-5. Extensive experiments demonstrate that ViT-5 consistently outperforms state-of-the-art plain Vision Transformers across both understanding and generation benchmarks. On ImageNet-1k classification, ViT-5-Base reaches 84.2\% top-1 accuracy under comparable compute, exceeding DeiT-III-Base at 83.8\%. ViT-5 also serves as a stronger backbone for generative modeling: when plugged into an SiT diffusion framework, it achieves 1.84 FID versus 2.06 with a vanilla ViT backbone. Beyond headline metrics, ViT-5 exhibits improved representation learning and favorable spatial reasoning behavior, and transfers reliably across tasks. With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s vision backbones.

Keywords

Cite

@article{arxiv.2602.08071,
  title  = {ViT-5: Vision Transformers for The Mid-2020s},
  author = {Feng Wang and Sucheng Ren and Tiezheng Zhang and Predrag Neskovic and Anand Bhattad and Cihang Xie and Alan Yuille},
  journal= {arXiv preprint arXiv:2602.08071},
  year   = {2026}
}

Comments

Code is available at https://github.com/wangf3014/ViT-5

R2 v1 2026-07-01T10:26:55.531Z