Despite extensive alignment efforts, Large Vision-Language Models (LVLMs) remain vulnerable to jailbreak attacks. To mitigate these risks, existing detection methods are essential, yet they face two major challenges: generalization and accuracy. While learning-based methods trained on specific attacks fail to generalize to unseen attacks, learning-free methods based on hand-crafted heuristics suffer from limited accuracy and reduced efficiency. To address these limitations, we propose Learning to Detect (LoD), a learnable framework that eliminates the need for any attack data or hand-crafted heuristics. LoD operates by first extracting layer-wise safety representations directly from the model's internal activations using Multi-modal Safety Concept Activation Vectors classifiers, and then converting the high-dimensional representations into a one-dimensional anomaly score for detection via a Safety Pattern Auto-Encoder. Extensive experiments demonstrate that LoD consistently achieves state-of-the-art detection performance (AUROC) across diverse unseen jailbreak attacks on multiple LVLMs, while also significantly improving efficiency. Code is available at https://anonymous.4open.science/r/Learning-to-Detect-51CB.
@article{arxiv.2508.09201,
title = {Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Models},
author = {Shuang Liang and Zhihao Xu and Jiaqi Weng and Jialing Tao and Hui Xue and Xiting Wang},
journal= {arXiv preprint arXiv:2508.09201},
year = {2026}
}
Comments
12 pages; Previously this version appeared as arXiv:2510.15430 which was submitted as a new work by accident