Transcriptomic foundation models pretrained with masked language modeling can achieve low pretraining loss yet produce poor cell representations for downstream tasks. We introduce whole-cell expression decoding (WCED), where models reconstruct the entire gene vocabulary from a single CLS token embedding, even with limited inputs, creating a maximally informative bottleneck. WCED consistently outperforms MLM on all downstream metrics despite higher reconstruction error during training. Gene-level error tracking reveals that both methods preferentially learn genes whose expression co-varies with stable transcriptional programs rather than those driven by transient factors. We further add hierarchical cross-entropy loss that exploits Cell Ontology structure for zero-shot annotation at multiple granularity levels. Models trained with these objectives achieve best overall performance across CZI benchmarks, on zero-shot batch integration and linear probing cell-type annotation. Methods are implemented in biomed-multi-omic ( https://github.com/BiomedSciAI/biomed-multi-omic ), an open-source framework for transcriptomic foundation model development.
@article{arxiv.2506.14861,
title = {BMFM-RNA: whole-cell expression decoding improves transcriptomic foundation models},
author = {Michael M. Danziger and Bharath Dandala and Viatcheslav Gurev and Matthew Madgwick and Sivan Ravid and Tim Rumbell and Akira Koseki and Tal Kozlovski and Ching-Huei Tsou and Ella Barkan and Tanwi Biswas and Jielin Xu and Yishai Shimoni and Jianying Hu and Michal Rosen-Zvi},
journal= {arXiv preprint arXiv:2506.14861},
year = {2026}
}