Referring expression comprehension (REC) is a vision-language task to locate a target object in an image based on a language expression. Fully fine-tuning general-purpose pre-trained vision-language foundation models for REC yields impressive performance but becomes increasingly costly. Parameter-efficient transfer learning (PETL) methods have shown strong performance with fewer tunable parameters. However, directly applying PETL to REC faces two challenges: (1) insufficient multi-modal interaction between pre-trained vision-language foundation models, and (2) high GPU memory usage due to gradients passing through the heavy vision-language foundation models. To this end, we present M2IST: Multi-Modal Interactive Side-Tuning with M3ISAs: Mixture of Multi-Modal Interactive Side-Adapters. During fine-tuning, we fix the pre-trained uni-modal encoders and update M3ISAs to enable efficient vision-language alignment for REC. Empirical results reveal that M2IST achieves better performance-efficiency trade-off than full fine-tuning and other PETL methods, requiring only 2.11\% tunable parameters, 39.61\% GPU memory, and 63.46\% training time while maintaining competitive performance. Our code is released at https://github.com/xuyang-liu16/M2IST.
@article{arxiv.2407.01131,
title = {M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension},
author = {Xuyang Liu and Ting Liu and Siteng Huang and Yi Xin and Yue Hu and Quanjun Yin and Donglin Wang and Yuanyuan Wu and Honggang Chen},
journal= {arXiv preprint arXiv:2407.01131},
year = {2025}
}
Comments
Accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)