Real-world clinical practice demands multi-image comparative reasoning, yet current medical benchmarks remain limited to single-frame interpretation. We present MedFrameQA, the first benchmark explicitly designed to test multi-image medical VQA through educationally-validated diagnostic sequences. To construct this dataset, we develop a scalable pipeline that leverages narrative transcripts from medical education videos to align visual frames with textual concepts, automatically producing 2,851 high-quality multi-image VQA pairs with explicit, transcript-grounded reasoning chains. Our evaluation of 11 advanced MLLMs (including reasoning models) exposes severe deficiencies in multi-image synthesis, where accuracies mostly fall below 50% and exhibit instability across varying image counts. Error analysis demonstrates that models often treat images as isolated instances, failing to track pathological progression or cross-reference anatomical shifts. MedFrameQA provides a rigorous standard for evaluating the next generation of MLLMs in handling complex, temporally grounded medical narratives.
@article{arxiv.2505.16964,
title = {MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning},
author = {Suhao Yu and Haojin Wang and Juncheng Wu and Luyang Luo and Jingshen Wang and Cihang Xie and Pranav Rajpurkar and Carl Yang and Yang Yang and Kang Wang and Yannan Yu and Yuyin Zhou},
journal= {arXiv preprint arXiv:2505.16964},
year = {2026}
}