Deep learning relies heavily on data augmentation to mitigate limited data, especially in medical imaging. Recent multimodal learning integrates text and images for segmentation, known as referring or text-guided image segmentation. However, common augmentations like rotation and flipping disrupt spatial alignment between image and text, weakening performance. To address this, we propose an early fusion framework that combines text and visual features before augmentation, preserving spatial consistency. We also design a lightweight generator that projects text embeddings into visual space, bridging semantic gaps. Visualization of generated pseudo-images shows accurate region localization. Our method is evaluated on three medical imaging tasks and four segmentation frameworks, achieving state-of-the-art results. Code is publicly available on GitHub: https://github.com/11yxk/MedSeg_EarlyFusion.
@article{arxiv.2510.12482,
title = {A Text-Image Fusion Method with Data Augmentation Capabilities for Referring Medical Image Segmentation},
author = {Shurong Chai and Rahul Kumar JAIN and Rui Xu and Shaocong Mo and Ruibo Hou and Shiyu Teng and Jiaqing Liu and Lanfen Lin and Yen-Wei Chen},
journal= {arXiv preprint arXiv:2510.12482},
year = {2025}
}