English

Transferability Between Understanding and Generation in Unified Multimodal Models

Computer Vision and Pattern Recognition 2026-07-05 v1 Artificial Intelligence

Abstract

Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate transferability\boldsymbol{\mathsf{transferability}} in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

Cite

@article{arxiv.2607.04423,
  title  = {Transferability Between Understanding and Generation in Unified Multimodal Models},
  author = {Jiwon Kang and Heeji Yoon and Jaewoo Jung and Jaewon Min and Minkyeong Jeon and Biyeon Hwang and Sangwon Jung and Seungryong Kim},
  journal= {arXiv preprint arXiv:2607.04423},
  year   = {2026}
}

Comments

Accepted at ECCV 2026. Project Page: this https URL