Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores
Abstract
Understanding complete musical scores entails integrated reasoning over pitch, rhythm, harmony, and large-scale structure, yet the ability of Large Language Models and Vision--Language Models to interpret full musical notation remains insufficiently examined. We introduce Musical Score Understanding Benchmark (MSU-Bench), a human-curated benchmark for score-level musical understanding across textual (ABC notation) and visual (PDF) modalities. MSU-Bench contains 1,800 generative question-answer pairs from works by Bach, Beethoven, Chopin, Debussy, and others, organised into four levels of increasing difficulty, ranging from onset information to texture and form. Evaluations of more than fifteen state-of-the-art models, in both zero-shot and fine-tuned settings, reveal pronounced modality gaps, unstable level-wise performance, and challenges in maintaining multilevel correctness. Fine-tuning substantially improves results across modalities while preserving general knowledge, positioning MSU-Bench as a robust foundation for future research in multimodal reasoning. The benchmark and code are available at https://github.com/Congren-Dai/MSU-Bench.
Keywords
Cite
@article{arxiv.2511.20697,
title = {Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores},
author = {Congren Dai and Yue Yang and Krinos Li and Huichi Zhou and Shijie Liang and Bo Zhang and Enyang Liu and Ge Jin and Hongran An and Haosen Zhang and Peiyuan Jing and Kinhei Lee and Z henxuan Zhang and Xiaobing Li and Maosong Sun},
journal= {arXiv preprint arXiv:2511.20697},
year = {2026}
}
Comments
Accepted to ACL 2026 Main Conference