The rapid development of Multimodal Large Language Models (MLLMs), such as GPT-4o, marks a significant step toward artificial general intelligence. Existing methods typically align vision encoders with LLMs via supervised fine-tuning (SFT), but this often deteriorates their ability to handle multiple languages as training progresses. We empirically observe that imbalanced SFT datasets, largely English-centric, degrade performance on non-English languages due to the failure in multilingual token alignment. To address this, we propose PARROT, a novel approach that leverages textual guidance for visual token alignment at the language level. PARROT conditions visual tokens on diverse language inputs and uses Mixture-of-Experts (MoE) to align multilingual tokens. By computing cross-attention between initial visual features and textual embeddings, we select the most relevant experts, converting visual tokens into language-specific representations. Additionally, we introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions, to assess multilingual capabilities. PARROT achieves state-of-the-art performance on both the multilingual benchmarks and a wide range of multimodal tasks. Code and dataset are available at: https://github.com/AIDC-AI/Parrot
@article{arxiv.2406.02539,
title = {Parrot: Multilingual Visual Instruction Tuning},
author = {Hai-Long Sun and Da-Wei Zhou and Yang Li and Shiyin Lu and Chao Yi and Qing-Guo Chen and Zhao Xu and Weihua Luo and Kaifu Zhang and De-Chuan Zhan and Han-Jia Ye},
journal= {arXiv preprint arXiv:2406.02539},
year = {2025}
}
Comments
Accepted to ICML 2025. Code and dataset are available at: https://github.com/AIDC-AI/Parrot