Recent advances in computational pathology have leveraged vision-language models to learn joint representations of Hematoxylin and Eosin (HE) images with spatial transcriptomic (ST) profiles. However, existing approaches typically align HE tiles with their corresponding ST profiles at a single scale, overlooking fine-grained cellular structures and their spatial organization. To address this, we propose Sigmma, a multi-modal contrastive alignment framework for learning hierarchical representations of HE images and spatial transcriptome profiles across multiple scales. Sigmma introduces multi-scale contrastive alignment, ensuring that representations learned at different scales remain coherent across modalities. Furthermore, by representing cell interactions as a graph and integrating inter- and intra-subgraph relationships, our approach effectively captures cell-cell interactions, ranging from fine to coarse, within the tissue microenvironment. We demonstrate that Sigmm learns representations that better capture cross-modal correspondences, leading to an improvement of avg. 9.78\% in the gene-expression prediction task and avg. 26.93\% in the cross-modal retrieval task across datasets. We further show that it learns meaningful multi-tissue organization in downstream analyses.
@article{arxiv.2511.15464,
title = {SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome},
author = {Dabin Jeong and Amirhossein Vahidi and Ciro Ramírez-Suástegui and Marie Moullet and Kevin Ly and Mohammad Vali Sanian and Sebastian Birk and Yinshui Chang and Adam Boxall and Daniyal Jafree and Lloyd Steele and Vijaya Baskar MS and Muzlifah Haniffa and Mohammad Lotfollahi},
journal= {arXiv preprint arXiv:2511.15464},
year = {2025}
}