Pretrained vision foundation models deliver strong performance across tasks with limited fine-tuning. However, their Vision Transformer (ViT) backbones impose high inference costs, limiting deployment on resource-constrained devices. In this work, we accelerate large-scale pretrained ViTs while preserving their feature extraction capabilities by exploiting the intrinsic convolution-like behavior of some attention heads. Specifically, we introduce an efficient depthwise convolution-based layer that serves as a drop-in replacement for these heads. Additionally, we propose simple strategies to identify which heads can be replaced and introduce a fine-tuning procedure that recovers downstream task performance. Across both image classification and segmentation tasks, our method achieves 17-20\% percent inference speedup with minimal performance degradation. We validate the approach through detailed derivations, extensive experiments, and efficiency benchmarks. The reference implementation is publicly available.
@article{arxiv.2605.22132,
title = {Accelerating Vision Foundation Models with Drop-in Depthwise Convolution},
author = {Carmelo Scribano and Mohammad Mahdi and Nedyalko Prisadnikov and Yuqian Fu and Giorgia Franchini and Danda Pani Paudel and Marko Bertogna and Luc Van Gool},
journal= {arXiv preprint arXiv:2605.22132},
year = {2026}
}