This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.
@article{arxiv.2512.22905,
title = {JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation},
author = {Kai Liu and Jungang Li and Yuchong Sun and Shengqiong Wu and Jianzhang Gao and Daoan Zhang and Wei Zhang and Sheng Jin and Sicheng Yu and Geng Zhan and Jiayi Ji and Fan Zhou and Liang Zheng and Shuicheng Yan and Hao Fei and Tat-Seng Chua},
journal= {arXiv preprint arXiv:2512.22905},
year = {2026}
}
Comments
Accepted by NeurIPS as a Spotlight paper. Code: https://github.com/JavisVerse/JavisGPT