ViT-5: Vision Transformers for The Mid-2020s
Abstract
This work presents a systematic investigation into modernizing Vision Transformer backbones by leveraging architectural advancements from the past five years. While preserving the canonical Attention-FFN structure, we conduct a component-wise refinement involving normalization, activation functions, positional encoding, gating mechanisms, and learnable tokens. These updates form a new generation of Vision Transformers, which we call ViT-5. Extensive experiments demonstrate that ViT-5 consistently outperforms state-of-the-art plain Vision Transformers across both understanding and generation benchmarks. On ImageNet-1k classification, ViT-5-Base reaches 84.2\% top-1 accuracy under comparable compute, exceeding DeiT-III-Base at 83.8\%. ViT-5 also serves as a stronger backbone for generative modeling: when plugged into an SiT diffusion framework, it achieves 1.84 FID versus 2.06 with a vanilla ViT backbone. Beyond headline metrics, ViT-5 exhibits improved representation learning and favorable spatial reasoning behavior, and transfers reliably across tasks. With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s vision backbones.
Keywords
Cite
@article{arxiv.2602.08071,
title = {ViT-5: Vision Transformers for The Mid-2020s},
author = {Feng Wang and Sucheng Ren and Tiezheng Zhang and Predrag Neskovic and Anand Bhattad and Cihang Xie and Alan Yuille},
journal= {arXiv preprint arXiv:2602.08071},
year = {2026}
}
Comments
Code is available at https://github.com/wangf3014/ViT-5