English

Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation

Computer Vision and Pattern Recognition 2024-01-15 v1 Artificial Intelligence
Authors: Chang Che , Qunwei Lin , Xinyu Zhao , Jiaxin Huang , Liqiang Yu
View on arXiv ↗ PDF ↗

Abstract

The process of transforming input images into corresponding textual explanations stands as a crucial and complex endeavor within the domains of computer vision and natural language processing. In this paper, we propose an innovative ensemble approach that harnesses the capabilities of Contrastive Language-Image Pretraining models.

Keywords

contrastive language-image pre-training vision-language models image captioning

Cite

@article{arxiv.2401.06167,
  title  = {Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation},
  author = {Chang Che and Qunwei Lin and Xinyu Zhao and Jiaxin Huang and Liqiang Yu},
  journal= {arXiv preprint arXiv:2401.06167},
  year   = {2024}
}

Related papers

View all related →
Computation and Language · Computer Science
CLIP also Understands Text: Prompting CLIP for Phrase Understanding
An Yan, Jiacheng Li, Wanrong Zhu, Yujie Lu +2
2022-10-13
Computer Vision and Pattern Recognition · Computer Science
Contrastive Language-Image Pre-Training with Knowledge Graphs
Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song +1
2022-10-18
Computer Vision and Pattern Recognition · Computer Science
VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts
Longtian Qiu, Renrui Zhang, Ziyu Guo, Ziyao Zeng +3
2023-08-11
Computer Vision and Pattern Recognition · Computer Science
Enhancing Image Retrieval : A Comprehensive Study on Photo Search using the CLIP Mode
Naresh Kumar Lahajal, Harini S
2024-01-25
Computer Vision and Pattern Recognition · Computer Science
Structure-aware Contrastive Learning for Diagram Understanding of Multimodal Models
Hiroshi Sasaki
2026-03-02
Computer Vision and Pattern Recognition · Computer Science
Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training
Haicheng Wang, Chen Ju, Weixiong Lin, Shuai Xiao +8
2024-12-03
Computer Vision and Pattern Recognition · Computer Science
Object-centric Binding in Contrastive Language-Image Pretraining
Rim Assouel, Pietro Astolfi, Florian Bordes, Michal Drozdzal +1
2025-02-21
Computer Vision and Pattern Recognition · Computer Science
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
Han Fang, Pengfei Xiong, Luhui Xu, Yu Chen
2021-06-22
Computer Vision and Pattern Recognition · Computer Science
CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han +6
2023-03-28
Computer Vision and Pattern Recognition · Computer Science
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen +3
2021-05-11
Computer Vision and Pattern Recognition · Computer Science
Contrastive Localized Language-Image Pre-Training
Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang +6
2025-02-20
Computer Vision and Pattern Recognition · Computer Science
Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
Alexandros Delitzas, Maria Parelli, Nikolas Hars, Georgios Vlassis +3
2023-06-12
Computer Vision and Pattern Recognition · Computer Science
DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents
Fuxiao Liu, Hao Tan, Chris Tensmeyer
2024-04-29
Computer Vision and Pattern Recognition · Computer Science
Turning a CLIP Model into a Scene Text Detector
Wenwen Yu, Yuliang Liu, Wei Hua, Deqiang Jiang +2
2023-03-28
Computer Vision and Pattern Recognition · Computer Science
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment
Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu +3
2023-03-03
Computer Vision and Pattern Recognition · Computer Science
Enhancing Vision Models for Text-Heavy Content Understanding and Interaction
Adithya TG, Adithya SK, Abhinav R Bharadwaj, Abhiram HA +1
2024-08-31
Computer Vision and Pattern Recognition · Computer Science
The Solution for Language-Enhanced Image New Category Discovery
Haonan Xu, Dian Chao, Xiangyu Wu, Zhonghua Wan +1
2024-07-09
Computer Vision and Pattern Recognition · Computer Science
CLIP in Medical Imaging: A Survey
Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang +7
2025-03-27
Computer Vision and Pattern Recognition · Computer Science
Exploring Visual Interpretability for Contrastive Language-Image Pre-training
Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu +1
2022-11-29
Computer Vision and Pattern Recognition · Computer Science
Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioning
Zhiyue Liu, Jinyuan Liu, Fanrong Ma
2023-12-15
Computation and Language · Computer Science
Jina CLIP: Your CLIP Model Is Also Your Text Retriever
Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang +10
2024-06-27
Computer Vision and Pattern Recognition · Computer Science
Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability
Zhiyu Zhu, Zhibo Jin, Jiayu Zhang, Nan Yang +3
2025-02-24
Computation and Language · Computer Science
SignCLIP: Connecting Text and Sign Language by Contrastive Learning
Zifan Jiang, Gerard Sant, Amit Moryossef, Mathias Müller +2
2024-10-08
Computer Vision and Pattern Recognition · Computer Science
PLIP: Language-Image Pre-training for Person Representation Learning
Jialong Zuo, Jiahao Hong, Feng Zhang, Changqian Yu +4
2024-05-30
Computer Vision and Pattern Recognition · Computer Science
CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
Maria Parelli, Alexandros Delitzas, Nikolas Hars, Georgios Vlassis +3
2023-04-14
R2 v1 2026-06-28T14:14:38.658Z