Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.
@article{arxiv.2405.17247,
title = {An Introduction to Vision-Language Modeling},
author = {Florian Bordes and Richard Yuanzhe Pang and Anurag Ajay and Alexander C. Li and Adrien Bardes and Suzanne Petryk and Oscar Mañas and Zhiqiu Lin and Anas Mahmoud and Bargav Jayaraman and Mark Ibrahim and Melissa Hall and Yunyang Xiong and Jonathan Lebensold and Candace Ross and Srihari Jayakumar and Chuan Guo and Diane Bouchacourt and Haider Al-Tahan and Karthik Padthe and Vasu Sharma and Hu Xu and Xiaoqing Ellen Tan and Megan Richards and Samuel Lavoie and Pietro Astolfi and Reyhane Askari Hemmat and Jun Chen and Kushal Tirumala and Rim Assouel and Mazda Moayeri and Arjang Talattof and Kamalika Chaudhuri and Zechun Liu and Xilun Chen and Quentin Garrido and Karen Ullrich and Aishwarya Agrawal and Kate Saenko and Asli Celikyilmaz and Vikas Chandra},
journal= {arXiv preprint arXiv:2405.17247},
year = {2024}
}