Covariance descriptors capture second-order statistics of image features. They have shown strong performance in general computer vision tasks, but remain underexplored in medical imaging. We investigate their effectiveness for both conventional and learning-based medical image classification, with a particular focus on SPDNet, a classification network specifically designed for symmetric positive definite (SPD) matrices. We propose constructing covariance descriptors from features extracted by pre-trained general vision encoders (GVEs) and comparing them with handcrafted descriptors. Two GVEs - DINOv2 and MedSAM - are evaluated across eleven binary and multi-class datasets from the MedMNSIT benchmark. Our results show that covariance descriptors derived from GVE features consistently outperform those derived from handcrafted features. Moreover, SPDNet yields superior performance to state-of-the-art methods when combined with DINOv2 features. Our findings highlight the potential of combining covariance descriptors with powerful pretrained vision encoders for medical image analysis.
@article{arxiv.2511.04190,
title = {Covariance Descriptors Meet General Vision Encoders: Riemannian Deep Learning for Medical Image Classification},
author = {Josef Mayr and Anna Reithmeir and Maxime Di Folco and Julia A. Schnabel},
journal= {arXiv preprint arXiv:2511.04190},
year = {2025}
}
Comments
Preprint. Submitted to the IEEE International Symposium on Biomedical Imaging (ISBI) 2026