In this paper, we introduce Modality-Inconsistent Continual Learning (MICL), a new continual learning scenario for Multimodal Large Language Models (MLLMs) that involves tasks with inconsistent modalities (image, audio, or video) and varying task types (captioning or question-answering). Unlike existing vision-only or modality-incremental settings, MICL combines modality and task type shifts, both of which drive catastrophic forgetting. To address these challenges, we propose MoInCL, which employs a Pseudo Targets Generation Module to mitigate forgetting caused by task type shifts in previously seen modalities. It also incorporates Instruction-based Knowledge Distillation to preserve the model's ability to handle previously learned modalities when new ones are introduced. We benchmark MICL using a total of six tasks and conduct experiments to validate the effectiveness of our MoInCL. The experimental results highlight the superiority of MoInCL, showing significant improvements over representative and state-of-the-art continual learning baselines.
@article{arxiv.2412.13050,
title = {Modality-Inconsistent Continual Learning of Multimodal Large Language Models},
author = {Weiguo Pian and Shijian Deng and Shentong Mo and Mingrui Liu and Yunhui Guo and Yapeng Tian},
journal= {arXiv preprint arXiv:2412.13050},
year = {2026}
}
Comments
Accepted at Transactions on Machine Learning Research (TMLR), 2026