Vision Language Models adapt well to downstream tasks but are highly vulnerable to adversarial perturbations that disrupt cross-modal semantic alignment. Existing defenses are largely unidirectional or structural, failing to exploit bidirectional cross-modal complementarity and instance-wise adaptive protection. To overcome the limitations of unidirectional and static defenses in adversarial settings, we propose Closed-Loop Bidirectional Prompting, casting robust adaptation as cross-modal agreement recovery via a dynamic feedback loop on frozen encoders. A Semantic Anchor is introduced as a stable prior to constrain cyclic updates and mitigate perturbation-induced feature corruption. Through anchor-based bootstrapping, textual semantics denoise visual representations, while the refined visuals enable instance-adaptive prompt updating, yielding a rectified and robust consensus. Extensive evaluations across 11 datasets validate state-of-the-art robustness and strong base-to-new generalization, while maintaining a favorable trade-off between computational cost and accuracy.
@article{arxiv.2605.25922,
title = {Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models},
author = {Xiao Liu and Jiaxiang Liu and Boci Peng and Boren Hu and Yusong Wang and Xiwen Chen and Prayag Tiwari and Liming Zhang and Mingkun Xu},
journal= {arXiv preprint arXiv:2605.25922},
year = {2026}
}