Despite recent progress in Multi-Modal Large Language Models (MLLMs), it remains challenging to integrate diverse tasks ranging from pixel-level perception to high-fidelity generation. Existing approaches often suffer from either restricted task extensibility or severe performance degradation due to modality interference. n this paper, we present LLMBind, an extensible framework that unifies multimodal tasks through a dual-pathway mechanism: In-Situ semantic embeddings for localization-sensitive tasks like semantic segmentation and Ex-Situ task-prompts for generation across image, video, and audio modalities. Additionally, we employ a Mixture-of-Experts (MoE) architecture to route task-specific tokens, thereby achieving modality disentanglement and mitigating negative transfer. We also curate a 400k multi-turn interactive dataset focused on iterative visual refinement to enable human-like interaction. Extensive experiments demonstrate that LLMBind achieves excellent performance across multiple perception and generation benchmarks while maintaining superior expandability.
@article{arxiv.2402.14891,
title = {LLMBind: A Unified Modality-Task Integration Framework},
author = {Bin Zhu and Munan Ning and Peng Jin and Bin Lin and Jinfa Huang and Qi Song and Junwu Zhang and Zhenyu Tang and Mingjun Pan and Li Yuan},
journal= {arXiv preprint arXiv:2402.14891},
year = {2026}
}