Skip to content

Single-Cell Foundation Models

Summary

Single-Cell Foundation Models (scFMs) are large-scale deep learning models (mostly based on transformer architectures) trained on massive single-cell datasets, typically using self-supervised objectives such as masked gene token prediction. However, recent systematic evaluation shows that scFMs display no clear data scaling laws, and their performance on downstream tasks (such as cell-type annotation, batch integration, and perturbation response prediction) often plateaus when pretrained on as little as 1% to 10% of their training corpora (approximately 200,000 cells). Furthermore, simpler baseline approaches like principal component analysis (PCA) or highly variable genes (HVGs) often perform comparably or better than these complex models.

Overview

Inspired by the success of large language models (LLMs), scFMs conceptualize a cell's transcriptome as a "sentence" where genes act as "tokens". They undergo pretraining on huge databases like CELLxGENE (often tens of millions of cells) to learn low-dimensional representation spaces (embeddings) of cellular states. These embeddings can then be used in zero-shot settings or fine-tuned for downstream tasks, including: - Cell-Type Classification: Labeling cells based on their transcriptional profiles. - Batch Integration: Mixing cells of the same type across different experimental batches or technologies while retaining biological differences. - Perturbation Response Prediction: Simulating the effect of genetic or chemical perturbations on gene expression.

Systematic Evaluation of Pretraining Scaling

In a landmark study, DenAdel et al. (2026) systematically pretrained 400 models across five different architectures to evaluate the effects of pretraining dataset size and diversity:

Models Evaluated

  1. scVI: A variational autoencoder representing deep generative models.
  2. Geneformer: A transformer-based model designed for network biology.
  3. SCimilarity: An autoencoder-based metric learning framework.
  4. Self-Supervised Learning (SSL): A masked autoencoder model.
  5. Pretrained PCA: A baseline linear projection method.

Downsampling Strategies

The pretraining datasets (sampled from the 22.2-million-cell scTab corpus [2]) were generated using: - Random Downsampling: Conserves natural cell-type proportions and transcriptomic diversity. - Cell-Type Re-Weighting: Aims to include equal proportions of each cell type. - Geometric Sketching: Downsamples by partitioning the transcriptomic space to capture transcriptional diversity without metadata.

Pretraining datasets were created at sizes of 1%, 10%, 25%, 50%, 75%, and 100% of the training corpus.

Performance Saturation and Scaling Laws

The evaluation of these models across 6,400 experiments revealed that: - No Data Scaling Laws: Unlike LLMs, where performance scales with data size, scFMs show no clear scaling benefits beyond a small fraction of the training data. - Plateau at Small Sizes: Model performance typically reaches a Learning Saturation Point at 1% to 10% of the training corpus (corresponding to less than 1 GB of data or ~200,000 cells). - Baselines vs. Foundation Models: In cell-type classification and batch integration, simpler baselines (PCA and HVG) or classical classifiers (logistic regression) often outperform or match complex models like Geneformer and SSL. SCimilarity was a notable exception, outperforming other approaches on cell-type classification. - No Benefit from Naive Diversity: Pretraining on re-weighted or geometrically-sketched datasets did not improve performance over random downsampling. - Spike-In Perturbation Data: Adding systematically perturbed cells (via genome-wide Perturb-seq data from Replogle et al. [3]) did not enhance downstream performance.

See Also

  • scBERT
  • scGPT
  • scFoundation
  • Universal Cell Embedding (UCE) — a zero-shot, cross-species foundation model using ESM2 protein embeddings to tokenize genes, avoiding orthology-set restrictions.
  • scCello — incorporates Cell Ontology structure into pretraining to improve generalization to unseen cell types, tissues, and donors.

  • TranscriptFormer — a generative, autoregressive foundation model trained across 12 species spanning 1.53 billion years of evolution, reporting stronger evolutionary-distance generalization than UCE.

  • Crawford Lab — contributor to systematic evaluation of single-cell foundation-model scaling.

  • Microsoft Research — research organization involved in the scaling evaluation.

Citations

[1] DenAdel, A. et al. (2026). Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance. Nature Methods, 23, 1447–1457. DOI: 10.1038/s41592-026-03120-y. Source paper: s41592-026-03120-y.pdf [2] Fischer, F. et al. (2024). sctab: scaling cross-tissue single-cell annotation models. Nat. Commun., 15, 6611. [3] Replogle, J. M. et al. (2022). Mapping information-rich genotype–phenotype landscapes with genome-scale Perturb-seq. Cell, 185, 2559–2575.