Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potential of a state-of-the-art Vision and Language model, VinVL, to caption images at the scene level using (1) a novel dataset which pairs images with both object-centric and scene descriptions. Through (2) an in-depth analysis of the effect of the fine-tuning, we show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene; the model acquires a more holistic view of the image compared to when object-centric descriptions are generated. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.
@article{arxiv.2211.04971,
title = {Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions},
author = {Michele Cafagna and Kees van Deemter and Albert Gatt},
journal= {arXiv preprint arXiv:2211.04971},
year = {2022}
}