Skip to content

scBERT

Summary

Yang, Wang, Wang, et al. (2022) present scBERT, a transformer-based model adapting BERT's pretrain-then-finetune paradigm for cell-type annotation of single-cell RNA-seq data. scBERT is pretrained self-supervised on large-scale unlabeled scRNA-seq data from PanglaoDB, then fine-tuned on labeled reference datasets for supervised cell-type classification, and is designed to also flag unassignable cells for novel cell-type discovery rather than forcing a nearest-known-class label. Across nine benchmark datasets (17 organs, >50 cell types, >500,000 cells) and multiple robustness tests, scBERT outperformed marker-gene-based, correlation-based, and other machine-learning cell-type annotation methods, with particularly large margins on class-imbalanced and cross-cohort/batch-effect scenarios.

Method

  • Performer encoder: uses the Performer (a linear-attention Transformer variant) rather than standard quadratic self-attention, enabling efficient processing of the full ~20,000-gene transcriptome per cell without restricting to a marker-gene subset.
  • Gene embeddings: gene identity embeddings are derived from gene2vec (capturing gene-gene co-expression similarity as a relative/positional embedding, since gene order itself is not meaningful, unlike word order in text), summed with binned, discretized expression-value embeddings for that gene in that cell.
  • Pretraining: self-supervised masked expression reconstruction on unlabeled scRNA-seq data from PanglaoDB.
  • Fine-tuning: the pretrained Performer encoder is retained; a 1D convolution layer plus classifier head is added and trained on a labeled reference dataset to predict cell types, with low-confidence predictions flagged as "unassigned" to support novel cell-type discovery.

Performance

  • Standard benchmark (9 datasets, 5-fold CV): scBERT surpassed marker-gene-based (SCINA, Garnett, scSorter), correlation-based (Seurat v4, SingleR, scmap, Cell_ID), and ML-based (SciBet, scNym) methods in accuracy and macro F1 on most datasets.
  • Zheng68K PBMC benchmark: with all genes as input, scBERT achieved F1=0.691, accuracy=0.759, versus the best competing method's F1=0.659, accuracy=0.704; on the specifically hard CD8+ cytotoxic vs. CD8+/CD45RA+ naive T cell distinction, scBERT reached F1=0.788 vs. 0.617 for the next-best method (P=9.0×10⁻⁵) and accuracy=0.801 vs. 0.724 (P=2.3×10⁻⁵).
  • Class-imbalance robustness: on four highly transcriptomically similar, imbalanced Zheng68K cell types, scBERT achieved accuracy=0.840, F1=0.826, with the lowest misclassification rate among compared methods (Seurat misidentified CD8+ cytotoxic T cells; SingleR misclassified all CD19+ B cells due to their rarity).
  • Cross-cohort/batch-effect robustness: using a leave-one-dataset-out strategy across four independently sequenced human pancreas datasets (different platforms: inDrop, CEL-seq2, SMART-seq2, SMARTer), scBERT achieved accuracy=0.992, versus scNym=0.904, SciBet=0.985, Seurat=0.984, SingleR=0.987 — correctly annotating >97–99% of cells per dataset without requiring explicit batch correction, unlike correlation-based methods like Seurat.

See Also

  • scGPT — a later generative transformer foundation model benchmarked against scBERT on cell-type annotation, where scGPT reported outperforming scBERT and TOSICA.

Citations

[1] Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2022). scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nature Machine Intelligence, 4, 852–866. [2] Key Evaluation: DenAdel, A. et al. (2026). Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance. Nature Methods, 23, 1447–1457. DOI: 10.1038/s41592-026-03120-y. Source paper: s41592-026-03120-y.pdf