Geneformer

Summary

Theodoris, Xiao, Chopra, et al. (2023) present Geneformer, a bidirectional transformer model pretrained on Genecorpus-30M (~30 million single-cell transcriptomes spanning a broad range of human tissues), designed to capture gene network dynamics via self-supervised pretraining and transfer that knowledge to downstream tasks with only limited task-specific labeled data — including in silico deletion of genes to predict disease-relevant network effects and identification of candidate therapeutic targets in cardiomyopathy, later experimentally validated. In a later systematic benchmarking study (DenAdel et al. 2026, as of 2026-06-09), Geneformer's performance on cell-type annotation and batch integration tasks was shown to saturate when pretrained on just 1% to 10% of the training corpus, with simple baseline methods often performing comparably — a critical caveat to the transfer-learning claims below.

Architecture and Training

Geneformer adopts the BERT (Bidirectional Encoder Representations from Transformers) framework: - Input representation: cells are represented as rank value encodings — genes ranked by expression within each cell (normalized against corpus-wide gene expression, deprioritizing ubiquitously highly-expressed housekeeping genes), rather than absolute expression values, and non-expressed genes are excluded. - Model size: six transformer encoder layers with self-attention and feed-forward layers, embedding genes into a 256-dimensional contextual space. - Pretraining: entirely self-supervised masked learning — 15% of genes in each transcriptome are masked, and the model predicts the masked gene's identity from the remaining rank-ordered context, requiring no labeled data and enabling training on Genecorpus-30M's full unlabeled scale.

Transfer Learning Results

• Context-aware embeddings: gene embeddings were robust to sequencing platform, tissue preservation method, and individual patient variability, but shifted appropriately with cellular context — e.g., in silico reprogramming a fibroblast transcriptome by adding OCT4, SOX2, KLF4, and MYC to the front of its rank-value encoding shifted the remaining genes' embeddings toward the iPSC state.
• Cell-type annotation: compared against XGBoost and deep-neural-network baselines trained from scratch per tissue, Geneformer's transfer-learning approach improved cell-type prediction accuracy and macro F1, with the performance gap widening as the number of cell-type classes increased.
• Gene dosage sensitivity (limited-data fine-tuning): fine-tuned on only 10,000 single-cell transcriptomes to distinguish dosage-sensitive from dosage-insensitive transcription factors, Geneformer achieved AUC=0.91±0.02, substantially outperforming SVM (0.75), random forest (0.72), logistic regression (0.65), and non-pretrained attention-based models of matched or varying depth (0.41–0.56) — with larger and more diverse pretraining corpuses consistently improving this downstream task despite identical fine-tuning data.
• In silico deletion for disease modeling: in silico deletion of genes associated with cardiomyocyte-pathology diseases (cardiomyopathy, structural heart disease) produced more deleterious shifts in cardiomyocyte embeddings than deletion of control cardiac-disease genes whose pathology occurs in non-cardiomyocyte cell types; this was validated against real patient cardiomyopathy expression data.
• Experimental validation: Geneformer-predicted dosage-sensitive gene TEAD4 was CRISPR-knocked out in iPSC-derived engineered cardiac microtissues, significantly reducing contractile stress relative to control (P<0.05, Wilcoxon) — direct wet-lab confirmation of a computationally prioritized therapeutic target candidate.

Benchmarking Caveat

In a systematic study evaluating Single-Cell Foundation Models under downsampled pretraining regimes, Geneformer reached its Learning Saturation Point at 1% of the pretraining data (~200,000 cells) on cell-type classification tasks. Its performance was often exceeded by simpler methods like Principal Component Analysis (PCA) or highly variable genes (HVGs) — this evaluation used generic classification benchmarks rather than the limited-data transfer-learning and in silico perturbation tasks emphasized in the original paper, so the two results are not necessarily in conflict, but together they suggest Geneformer's demonstrated value is more specific to context-aware network-biology tasks than to general-purpose cell-type classification.

Citations

[1] Theodoris, C.V., Xiao, L., Chopra, A., Chaffin, M.D., Al Sayed, Z.R., Hill, M.C., Mantineo, H., Brydon, E.M., Zeng, Z., Liu, X.S., & Ellinor, P.T. (2023). Transfer learning enables predictions in network biology. Nature, 618, 616–624. [2] Key Evaluation: DenAdel, A. et al. (2026). Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance. Nature Methods, 23, 1447–1457. DOI: 10.1038/s41592-026-03120-y. Source paper: s41592-026-03120-y.pdf