English

ViTOC: Vision Transformer and Object-aware Captioner

Computer Vision and Pattern Recognition 2025-04-18 v4

Abstract

This paper presents ViTOC (Vision Transformer and Object-aware Captioner), a novel vision-language model for image captioning that addresses the challenges of accuracy and diversity in generated descriptions. Unlike conventional approaches, ViTOC employs a dual-path architecture based on Vision Transformer and object detector, effectively fusing global visual features and local object information through learnable vectors. The model introduces an innovative object-aware prompting strategy that significantly enhances its capability in handling long-tail data. Experiments on the standard COCO dataset demonstrate that ViTOC outperforms baseline models across all evaluation metrics. Additionally, we propose a reference-free evaluation method based on CLIP to further validate the model's effectiveness. By utilizing pretrained visual model parameters, ViTOC achieves efficient end-to-end training.

Keywords

Cite

@article{arxiv.2411.07265,
  title  = {ViTOC: Vision Transformer and Object-aware Captioner},
  author = {Feiyang Huang},
  journal= {arXiv preprint arXiv:2411.07265},
  year   = {2025}
}

Comments

The core idea is too close to what has been published in other journals

R2 v1 2026-06-28T19:55:58.190Z