scGPT

Summary

Cui, Wang, Maan, et al. (2024) present scGPT, a generative foundation model for single-cell genomics pretrained on over 33 million normal human cells from the CELLxGENE collection (51 organs/tissues, 441 studies), using a generative transformer that simultaneously learns gene and cell representations. Fine-tuned on downstream tasks, scGPT reports state-of-the-art or competitive performance across cell-type annotation, batch integration, multi-omic integration, genetic perturbation response prediction, and gene regulatory network inference, and the paper reports a scaling effect where larger pretraining corpora consistently improve downstream performance.

Architecture

scGPT leverages generative pretraining: - Input embedding: each cell's input combines gene tokens, expression values, and condition tokens (modality, batch, perturbation status). - Specialized generative attention: a custom attention mask in the transformer's masked multi-head attention blocks enables generative (rather than purely discriminative/masked-reconstruction) pretraining directly on non-sequential single-cell expression data. - Dual embeddings: learns joint embedding spaces for both individual genes and whole cells. - Multimodal capabilities: extensible to capture chromatin accessibility (scATAC-seq) and protein abundance (CITE-seq) alongside transcriptomic data.

Downstream Performance

• Cell-type annotation: on a human pancreas dataset, fine-tuned scGPT achieved >0.8 precision for most cell types (weakest on extremely rare types, e.g., <50 reference cells); on a multiple sclerosis immune-cell dataset, ~0.85 accuracy predicting disease-state cell types from a healthy-only reference; on a tumor-infiltrating myeloid dataset generalizing across six training cancer types to three unseen types, scGPT consistently outperformed TOSICA and scBERT on accuracy, precision, recall, and macro F1.
• Genetic perturbation response prediction: evaluated on three Perturb-seq datasets (Adamson: 87 one-gene perturbations; Replogle: 1,823 one-gene perturbations; Norman: 131 two-gene + 105 one-gene perturbations), fine-tuned scGPT predicted post-perturbation expression changes for unseen perturbed genes, outperforming GEARS and a linear-regression baseline by 5–20% margins on the Pearson-delta correlation metric (including on the top-20 differentially expressed genes per perturbation).
• Scaling effect: larger and more diverse pretraining datasets consistently produced better pretrained embeddings and downstream task performance, motivating continued corpus growth as a lever for model improvement.

Benchmarking Caveat

A later independent benchmarking study (DenAdel et al. 2026) found that across several single-cell foundation models including scGPT, cell-type annotation and batch-integration performance saturated with only 1–10% of the full pretraining corpus, and simple baselines (PCA, highly variable genes) were often competitive — see Geneformer's page for the same caveat, which applies across this model family and should be weighed against the task-specific performance claims above.

Citations

[1] Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., Duan, N., & Wang, B. (2024). scGPT: toward building a foundation model for single-cell multi-omics using generative AI. Nature Methods, 21, 1470–1480. [2] Key Evaluation: DenAdel, A. et al. (2026). Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance. Nature Methods, 23, 1447–1457. DOI: 10.1038/s41592-026-03120-y. Source paper: s41592-026-03120-y.pdf