We present a transformer-based multimodal framework for generating clinically relevant captions for MRI scans. Our system combines a DEiT-Small vision transformer as an image encoder, MediCareBERT for caption embedding, and a custom LSTM-based decoder. The architecture is designed to semantically align image and textual embeddings, using hybrid cosine-MSE loss and contrastive inference via vector similarity. We benchmark our method on the MultiCaRe dataset, comparing performance on filtered brain-only MRIs versus general MRI images against state-of-the-art medical image captioning methods including BLIP, R2GenGPT, and recent transformer-based approaches. Results show that focusing on domain-specific data improves caption accuracy and semantic alignment. Our work proposes a scalable, interpretable solution for automated medical image reporting.
@article{arxiv.2510.25164,
title = {Transformers in Medicine: Improving Vision-Language Alignment for Medical Image Captioning},
author = {Yogesh Thakku Suresh and Vishwajeet Shivaji Hogale and Luca-Alexandru Zamfira and Anandavardhana Hegde},
journal= {arXiv preprint arXiv:2510.25164},
year = {2025}
}
Comments
This work is to appear in the Proceedings of MICAD 2025, the 6th International Conference on Medical Imaging and Computer-Aided Diagnosis