Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space
@article{arxiv.2109.04993,
title = {LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation},
author = {Mohammad Abuzar Hashemi and Zhanghexuan Li and Mihir Chauhan and Yan Shen and Abhishek Satbhai and Mir Basheer Ali and Mingchen Gao and Sargur Srihari},
journal= {arXiv preprint arXiv:2109.04993},
year = {2024}
}
Comments
15 pages, 10 Figures, 5 Tables. Accepted for Oral Presentation at Irish Machine Vision and Image Processing Conference Proceedings (IMVIP), 2024