Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection

Subaru Kimura; Ryota Tanaka; Shumpei Miyawaki; Jun Suzuki; Keisuke Sakaguchi

Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection

Computation and Language 2024-08-08 v1 Cryptography and Security Machine Learning

Authors: Subaru Kimura , Ryota Tanaka , Shumpei Miyawaki , Jun Suzuki , Keisuke Sakaguchi

Abstract

We explore visual prompt injection (VPI) that maliciously exploits the ability of large vision-language models (LVLMs) to follow instructions drawn onto the input image. We propose a new VPI method, "goal hijacking via visual prompt injection" (GHVPI), that swaps the execution task of LVLMs from an original task to an alternative task designated by an attacker. The quantitative analysis indicates that GPT-4V is vulnerable to the GHVPI and demonstrates a notable attack success rate of 15.8%, which is an unignorable security risk. Our analysis also shows that successful GHVPI requires high character recognition capability and instruction-following ability in LVLMs.

Keywords

vision-language models prompt injection security adversarial attack large language models

Cite

@article{arxiv.2408.03554,
  title  = {Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection},
  author = {Subaru Kimura and Ryota Tanaka and Shumpei Miyawaki and Jun Suzuki and Keisuke Sakaguchi},
  journal= {arXiv preprint arXiv:2408.03554},
  year   = {2024}
}

Comments

8 pages, 6 figures, Accepted to NAACL 2024 SRW

Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection

Abstract

Keywords

Cite

Comments

Related papers