Efficient transfer learning methods for large-scale vision-language models (e.g., CLIP) enable strong few-shot transfer, yet existing adaptation methods follow a fixed fine-tuning paradigm that implicitly assumes a uniform importance of the image and text branches, which has not been systematically studied in image classification. Through extensive analysis, we reveal a Branch Bias issue in vision-language image classification: adapting the image encoder does not always improve performance under out-of-distribution settings. Motivated by this observation, we propose A3B2, an Adaptive Asymmetric Adapter that alleviates Branch Bias in few-shot learning. A3B2 introduces Uncertainty-Aware Adapter Dampening (UAAD), which automatically suppresses image-branch adaptation when prediction uncertainty is high, enabling soft and data-driven control without manual intervention. Architecturally, A3B2 adopts a lightweight asymmetric design inspired by mixture-of-experts with Load Balancing Regularization. Extensive experiments on three few-shot image classification tasks across 11 datasets demonstrate that A3B2 consistently outperforms 11 competitive prompt- and adapter-based baselines.
@article{arxiv.2605.13161,
title = {A$_3$B$_2$: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning},
author = {Yiyun Zhou and Zhonghua Jiang and Wenkang Han and Kunxi Li and Mingjing Xu and Chang Yao and Jingyuan Chen},
journal= {arXiv preprint arXiv:2605.13161},
year = {2026}
}