TranscriptFormer
Summary¶
TranscriptFormer is a family of generative, autoregressive single-cell foundation models trained on up to 112 million cells spanning 1.53 billion years of evolution across 12 species (vertebrates, invertebrates, a fungus, and a protist). Like UCE, it maps genes into a shared, species-agnostic embedding space using ESM2 protein-sequence embeddings, avoiding the shrinking-ortholog-set problem that limits comparative analysis across distantly related species — but unlike UCE and other prior single-cell foundation models, TranscriptFormer is generative (modeling the joint probability of gene identities and expression levels autoregressively over a "cell sentence"), enabling zero-shot prediction and model "prompting" for tasks like transcription-factor-target prediction or perturbation-response simulation without task-specific fine-tuning.
Model Variants and Architecture¶
- TF-Metazoa: 112M cells, 12 species (6 vertebrates: human, mouse, rabbit, chicken, African clawed frog, zebrafish; 4 invertebrates: green sea urchin, roundworm, fruit fly, freshwater sponge; 2 outgroups: baker's yeast, malaria parasite), spanning 1.53 billion years of divergence.
- TF-Exemplar: 110M cells, human + 4 model organisms (mouse, zebrafish, fruit fly, roundworm).
- TF-Sapiens: 57M human-only cells.
- All three variants share the same architecture and active parameter count, with training cell counts fixed and species-specific sampling rates balanced for fair comparison.
- Architecture: transformer coupling separate gene and transcript prediction heads, expression-aware multi-head self-attention, causal masking (autoregressive generation), and a count likelihood modeling transcript-level variability; genes are tokenized via ESM2 protein embeddings (as in UCE), giving a species-agnostic shared gene representation.
Performance¶
- Out-of-distribution species generalization: on cell atlases from species never seen during training (mouse lemur, tropical clawed frog, sea lamprey, stony coral), TranscriptFormer variants maintained macro F1=0.65–0.91 even for stony coral (685 million years diverged from humans), while UCE dropped to F1≤0.5 for distant species; both TranscriptFormer variants also outperformed an ESM2-embedding-only baseline (ESM2-CE) across all species, showing the value of training on actual transcriptomic data rather than protein sequence alone.
- In-distribution (human/mouse) benchmark: TF-Metazoa achieved the highest average F1 (0.778±0.002), ahead of TF-Exemplar (0.770±0.002), UCE (0.701±0.002), and GeneCompass.
- Cross-species annotation transfer (spermatogenesis, 9 vertebrate species): TF-Metazoa achieved F1=0.80 for marmoset-to-rhesus cell-type transfer (near self-transfer scores of 0.85/0.86); TF-Exemplar led with a species-averaged F1=0.480±0.002 versus UCE's 0.377±0.002; for chicken-to-mammal transfer (310 million years divergence), TF-Exemplar scored 0.448±0.006 versus UCE's 0.251±0.004, and TF-Exemplar improved 91% over the ESM2-CE baseline.
- Cross-species transfer, additional tissues: for primate dorsolateral prefrontal cortex, TF-Exemplar again led (0.744±0.002 average transfer score), followed by TF-Metazoa (0.712±0.002) and UCE (0.55±0.002).
- Inflammatory response transfer: LPS-perturbation annotation transfer in bone-marrow-derived phagocytes succeeded across mouse, rat, rabbit, and pig with F1 scores exceeding 0.92.
- Zero-shot disease/perturbation detection: without task-specific training, TranscriptFormer accurately identified SARS-CoV-2 infection status across donors and detected drug-induced transcriptional changes across 95 compounds in the Tahoe-100M perturbation dataset, substantially outperforming benchmark models.
- Ablation: controlled experiments at matched cell counts confirmed that adding phylogenetic diversity to the training corpus substantially improved cross-species generalization without degrading in-distribution (human/mouse) performance.
- Emergent structure: developmental trajectories and phylogenetic relationships emerged naturally in the learned embedding space without explicit training on these annotations.
See Also¶
- UCE — the primary comparator throughout, sharing the ESM2-protein-embedding gene-tokenization strategy but using a non-generative (discriminative embedding) rather than autoregressive generative architecture; TranscriptFormer consistently outperforms UCE on evolutionarily distant cross-species tasks.
- scCello — another recent single-cell foundation model incorporating external biological structure (cell ontology rather than cross-species phylogeny) into pretraining.
- Geneformer, scGPT — earlier, human-centric single-cell foundation models also used as baselines.