Research on large language models has advanced significantly across text, speech, images, and videos. However, multi-modal music understanding and generation remain underexplored due to the lack of well-annotated datasets. To address this, we introduce a dataset with 167.69 hours of multi-modal data, including text, images, videos, and music annotations. Based on this dataset, we propose MuMu-LLaMA, a model that leverages pre-trained encoders for music, images, and videos. For music generation, we integrate AudioLDM 2 and MusicGen. Our evaluation across four tasks--music understanding, text-to-music generation, prompt-based music editing, and multi-modal music generation--demonstrates that MuMu-LLaMA outperforms state-of-the-art models, showing its potential for multi-modal music applications.
@article{arxiv.2412.06660,
title = {MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models},
author = {Shansong Liu and Atin Sakkeer Hussain and Qilong Wu and Chenshuo Sun and Ying Shan},
journal= {arXiv preprint arXiv:2412.06660},
year = {2024}
}