Large Vision-Language Models (LVLMs) have demonstrated impressive performance on multimodal tasks through scaled architectures and extensive training. However, existing Mixture of Experts (MoE) approaches face challenges due to the asymmetry between visual and linguistic processing. Visual information is spatially complete, while language requires maintaining sequential context. As a result, MoE models struggle to balance modality-specific features and cross-modal interactions. Through systematic analysis, we observe that language experts in deeper layers progressively lose contextual grounding and rely more on parametric knowledge rather than utilizing the provided visual and linguistic information. To address this, we propose AsyMoE, a novel architecture that models this asymmetry using three specialized expert groups. We design intra-modality experts for modality-specific processing, hyperbolic inter-modality experts for hierarchical cross-modal interactions, and evidence-priority language experts to suppress parametric biases and maintain contextual grounding. Extensive experiments demonstrate that AsyMoE achieves 26.58% and 15.45% accuracy improvements over vanilla MoE and modality-specific MoE respectively, with 25.45% fewer activated parameters than dense models.
@article{arxiv.2509.12715,
title = {AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models},
author = {Heng Zhang and Haichuan Hu and Yaomin Shen and Weihao Yu and Yilei Yuan and Haochen You and Guo Cheng and Zijian Zhang and Lubin Gan and Huihui Wei and Hao Zhang and Jin Huang},
journal= {arXiv preprint arXiv:2509.12715},
year = {2025}
}
Comments
This submission has been withdrawn by the authors due to a fundamental error in the methodology that affects the validity of the main results