With the advancement of Large Language Model (LLM) for natural language processing, this paper presents an intriguing finding: a frozen pre-trained LLM layer can process visual tokens for medical image segmentation tasks. Specifically, we propose a simple hybrid structure that integrates a pre-trained, frozen LLM layer within the CNN encoder-decoder segmentation framework (LLM4Seg). Surprisingly, this design improves segmentation performance with a minimal increase in trainable parameters across various modalities, including ultrasound, dermoscopy, polypscopy, and CT scans. Our in-depth analysis reveals the potential of transferring LLM's semantic awareness to enhance segmentation tasks, offering both improved global understanding and better local modeling capabilities. The improvement proves robust across different LLMs, validated using LLaMA and DeepSeek.
@article{arxiv.2506.18034,
title = {Pre-Trained LLM is a Semantic-Aware and Generalizable Segmentation Booster},
author = {Fenghe Tang and Wenxin Ma and Zhiyang He and Xiaodong Tao and Zihang Jiang and S. Kevin Zhou},
journal= {arXiv preprint arXiv:2506.18034},
year = {2025}
}
Comments
Accepted by MICCAI 2025. Code: https://github.com/FengheTan9/LLM4Seg