Large Language Models can Share Images, Too!
Abstract
This paper explores the image-sharing capability of Large Language Models (LLMs), such as GPT-4 and LLaMA 2, in a zero-shot setting. To facilitate a comprehensive evaluation of LLMs, we introduce the PhotoChat++ dataset, which includes enriched annotations (i.e., intent, triggering sentence, image description, and salient information). Furthermore, we present the gradient-free and extensible Decide, Describe, and Retrieve (DribeR) framework. With extensive experiments, we unlock the image-sharing capability of DribeR equipped with LLMs in zero-shot prompting, with ChatGPT achieving the best performance. Our findings also reveal the emergent image-sharing ability in LLMs under zero-shot conditions, validating the effectiveness of DribeR. We use this framework to demonstrate its practicality and effectiveness in two real-world scenarios: (1) human-bot interaction and (2) dataset augmentation. To the best of our knowledge, this is the first study to assess the image-sharing ability of various LLMs in a zero-shot setting. We make our source code and dataset publicly available at https://github.com/passing2961/DribeR.
Keywords
Cite
@article{arxiv.2310.14804,
title = {Large Language Models can Share Images, Too!},
author = {Young-Jun Lee and Dokyong Lee and Joo Won Sung and Jonghwan Hyeon and Ho-Jin Choi},
journal= {arXiv preprint arXiv:2310.14804},
year = {2024}
}
Comments
ACL 2024 Findings; Code is available in https://github.com/passing2961/DribeR