English

Semantic Generative Tuning for Unified Multimodal Models

Computer Vision and Pattern Recognition 2026-05-19 v1 Artificial Intelligence

Abstract

Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture. However, prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives. Such a decoupled strategy yields misaligned representation spaces, isolating visual understanding from generation and hindering their mutual reinforcement. This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs. Our empirical investigation reveals that high-level semantic tasks, particularly image segmentation, serve as optimal proxies. Unlike low-level tasks that distract models with texture details, segmentation provides structural semantics that significantly enhance both vision-centric perception and generative layout fidelity. Building upon these insights, we introduce Semantic Generative Tuning (SGT), a novel paradigm that leverages segmentation as a generative proxy to align and synergize multimodal capabilities. Mechanistic analyses further demonstrate that SGT fundamentally improves feature linear separability and optimizes visual-textual attention allocation pattern. Extensive evaluations show that SGT consistently improves both multimodal comprehension and generative fidelity across mainstream benchmarks. Our code is available on the https://song2yu.github.io/SGT/.

Keywords

Cite

@article{arxiv.2605.18714,
  title  = {Semantic Generative Tuning for Unified Multimodal Models},
  author = {Songsong Yu and Yuxin Chen and Ying Shan and Yanwei Li},
  journal= {arXiv preprint arXiv:2605.18714},
  year   = {2026}
}

Comments

14 pages, 13 figures