Visual prompting has gained popularity as a method for adapting pre-trained models to specific tasks, particularly in the realm of parameter-efficient tuning. However, existing visual prompting techniques often pad the prompt parameters around the image, limiting the interaction between the visual prompts and the original image to a small set of patches while neglecting the inductive bias present in shared information across different patches. In this study, we conduct a thorough preliminary investigation to identify and address these limitations. We propose a novel visual prompt design, introducing Low-Rank matrix multiplication for Visual Prompting (LoR-VP), which enables shared and patch-specific information across rows and columns of image pixels. Extensive experiments across seven network architectures and four datasets demonstrate significant improvements in both performance and efficiency compared to state-of-the-art visual prompting methods, achieving up to 6 times faster training times, utilizing 18 times fewer visual prompt parameters, and delivering a 3.1% improvement in performance. The code is available as https://github.com/jincan333/LoR-VP.
@article{arxiv.2502.00896,
title = {LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation},
author = {Can Jin and Ying Li and Mingyu Zhao and Shiyu Zhao and Zhenting Wang and Xiaoxiao He and Ligong Han and Tong Che and Dimitris N. Metaxas},
journal= {arXiv preprint arXiv:2502.00896},
year = {2025}
}