Evaluating scientific arguments requires assessing the strict consistency between a claim and its underlying multimodal evidence. However, existing benchmarks lack the scale, domain diversity, and visual complexity needed to evaluate this alignment realistically. To address this gap, we introduce M2-Verify, a large-scale multimodal dataset for checking scientific claim consistency. Sourced from PubMed and arXiv, M2-Verify provides over 469K instances across 16 domains, rigorously validated through expert audits. Extensive baseline experiments show that state-of-the-art models struggle to maintain robust consistency. While top models achieve up to 85.8\% Micro-F1 on low-complexity medical perturbations, performance drops to 61.6\% on high-complexity challenges like anatomical shifts. Furthermore, expert evaluations expose hallucinations when models generate scientific explanations for their alignment decisions. Finally, we demonstrate our dataset's utility and provide comprehensive usage guidelines.
@article{arxiv.2604.01306,
title = {M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency},
author = {Abolfazl Ansari and Delvin Ce Zhang and Zhuoyang Zou and Wenpeng Yin and Dongwon Lee},
journal= {arXiv preprint arXiv:2604.01306},
year = {2026}
}