Skip to content

scFoundation

Summary

Hao, Gong, Zeng, et al. (2024) present scFoundation (internally, "xTrimoscFoundation-α"), a ~100-million-parameter foundation model covering ~20,000 (19,264) genes, pretrained on over 50 million human single-cell transcriptomic profiles spanning 100+ tissue types across disease, tumor, and normal states — one of the largest single-cell foundation models by parameter count, gene coverage, and pretraining data volume at publication. Its asymmetric transformer encoder-decoder architecture (xTrimoGene) and a novel "read-depth-aware" (RDA) pretraining task are designed to jointly model gene co-expression and normalize away sequencing read-depth variation, and the model reports state-of-the-art performance on gene expression enhancement, bulk/single-cell drug response prediction, single-cell perturbation prediction, cell-type annotation, and gene module inference.

Architecture and Pretraining

  • xTrimoGene architecture: an embedding module converts continuous (not discretized) gene expression scalars into learnable high-dimensional vectors, preserving raw expression information more faithfully than prior models' binned/discretized inputs; an asymmetric encoder-decoder (in the spirit of masked autoencoders from computer vision) is designed to efficiently handle scRNA-seq data's high sparsity without needing to restrict to a subset of genes.
  • Read-depth-aware (RDA) pretraining: extending masked language modeling, the model is given a (possibly downsampled) "source" version of a cell's profile with target ("T") and source ("S") total-count indicators, and learns to reconstruct the raw ("target") expression at masked positions — this both teaches gene-gene relationships and lets the model be used at inference to "enhance" a low-read-depth cell by setting T higher than its actual total count S.
  • Scale: 100M parameters, 19,264 genes, >50M pretraining cells from GEO, HCA, EMBL-EBI, hECA, DISCO and other public single-cell resources.

Performance

  • Scaling law: training 3M, 10M, and 100M-parameter versions showed a power-law decline in validation loss with increasing parameters/compute, and cell-type annotation performance improved with model size — the 100M-parameter model surpassed all smaller-scale comparisons, including parameter-matched reimplementations of prior transformer-based scFMs and scVI.
  • Read-depth enhancement: on held-out cells downsampled to 1–20% of original total counts, non-fine-tuned scFoundation halved the mean absolute error and mean relative error of recovered nonzero gene expression relative to the downsampled input, even below 10% downsampling.
  • Clustering after imputation: compared to MAGIC, SAVER, scImpute, and scVI on a downsampled human pancreatic islet benchmark (evaluated via NMI, ARI, silhouette score against ground-truth reference clusters), scFoundation's performance improved sharply as the read-depth-enhancement fold (T/S) increased, surpassing all other methods and plateauing above ~3.5×; it was the only method whose cluster assignments consistently aligned with the reference UMAP across all clusters.
  • Cell-type annotation benchmark: against CellTypist, scBERT, scANVI, ACTINN, Scanpy, and SingleCellNet, scFoundation achieved the highest macro F1 score on both evaluation datasets.
  • Batch-effect mitigation: feeding read-depth-enhanced cell embeddings into a non-trainable BBKNN downstream header improved cross-batch cell mapping on simulated and real (organoid, in vivo) data relative to using raw expression, while only slightly reducing separation between distinct cell types — the authors note read-depth harmonization alone does not eliminate all batch-effect sources (e.g., donor sex, treatment, cell cycle).

See Also

  • scBERT — an earlier cell-type-annotation-focused transformer scFoundation directly outperformed in macro F1 benchmarking.

Citations

[1] Hao, M., Gong, J., Zeng, X., Liu, C., Guo, Y., Cheng, X., Wang, T., Ma, J., Zhang, X., & Song, L. (2024). Large-scale foundation model on single-cell transcriptomics. Nature Methods, 21, 1481–1491. [2] Key Evaluation: DenAdel, A. et al. (2026). Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance. Nature Methods, 23, 1447–1457. DOI: 10.1038/s41592-026-03120-y. Source paper: s41592-026-03120-y.pdf