Human-interpretable predictions are essential for deploying AI in medical imaging, yet most interpretable-by-design (IBD) frameworks require concept annotations for training data, which are costly and impractical to obtain in clinical contexts. Recent attempts to bypass annotation, such as zero-shot vision-language models or concept-generation frameworks, struggle to capture domain-specific medical features, leading to poor reliability. In this paper, we propose a novel Prior-guided Concept Predictor (PCP), a weakly supervised framework that enables concept answer prediction without explicit supervision or reliance on language models. PCP leverages class-level concept priors as weak supervision and incorporates a refinement mechanism with KL divergence and entropy regularization to align predictions with clinical reasoning. Experiments on PH2 (dermoscopy) and WBCatt (hematology) show that PCP improves concept-level F1-score by over 33% compared to zero-shot baselines, while delivering competitive classification performance on four medical datasets (PH2, WBCatt, HAM10000, and CXR4) relative to fully supervised concept bottleneck models (CBMs) and V-IP.
@article{arxiv.2511.01131,
title = {Weakly Supervised Concept Learning with Class-Level Priors for Interpretable Medical Diagnosis},
author = {Md Nahiduzzaman and Steven Korevaar and Alireza Bab-Hadiashar and Ruwan Tennakoon},
journal= {arXiv preprint arXiv:2511.01131},
year = {2026}
}
Comments
Accepted to IEEE International Symposium on Biomedical Imaging (ISBI) 2026