While emerging 3D medical foundation models are envisioned as versatile tools with offer general-purpose capabilities, their validation remains largely confined to regional and structural imaging, leaving a significant modality discrepancy unexplored. To provide a rigorous and objective assessment, we curate the UMD dataset comprising 490 whole-body PET/CT and 464 whole-body PET/MRI scans (∼675k 2D images, ∼12k 3D organ annotations) and conduct a thorough and comprehensive evaluation of representative 3D segmentation foundation models. Through intra-subject controlled comparisons of paired scans, we isolate imaging modality as the primary independent variable to evaluate model robustness in real-world applications. Our evaluation reveals a stark discrepancy between literature-reported benchmarks and real-world efficacy, particularly when transitioning from structural to functional domains. Such systemic failures underscore that current 3D foundation models are far from achieving truly general-purpose status, necessitating a paradigm shift toward multi-modal training and evaluation to bridge the gap between idealized benchmarking and comprehensive clinical utility. This dataset and analysis establish a foundational cornerstone for future research to develop truly modality-agnostic medical foundation models.
@article{arxiv.2602.07643,
title = {Uncovering Modality Discrepancy and Generalization Illusion for General-Purpose 3D Medical Segmentation},
author = {Yichi Zhang and Feiyang Xiao and Le Xue and Wenbo Zhang and Gang Feng and Chenguang Zheng and Yuan Qi and Yuan Cheng and Zixin Hu},
journal= {arXiv preprint arXiv:2602.07643},
year = {2026}
}