Skip to content

Enformer

Summary

Enformer — a portmanteau of enhancer and transformer — predicts 5,313 human genomic tracks (CAGE, DNase, histone ChIP-seq, TF ChIP-seq) at 128 bp resolution from 200 kb of input DNA, with self-attention giving it a 100 kb receptive field where the previous state of the art reached only 20 kb.[1] That single change is the paper's thesis: replacing dilated convolutions with attention lets information flow between distal elements, and the proportion of high-confidence enhancer–gene pairs the model can even see rises from 47% to 84%.[1]

It is the architectural common ancestor of the two models that superseded it — Borzoi reuses its convolution-and-attention trunk, and AlphaGenome shares three of its authors (Avsec, Taylor, Kohli) and its overall framing.

Architecture

  • Input 200 kb of DNA; output 5,313 human and 1,643 mouse tracks at 128 bp bins.[1]
  • A convolution tower followed by transformer blocks with custom relative positional basis functions, chosen so the model can distinguish proximal from distal elements and upstream from downstream of the TSS.[1]

The ablations are the most transferable part of the paper:[1]

  • Attention layers beat dilated convolutions across all model sizes, layer counts and training-data volumes.
  • Restricting Enformer's receptive field to Basenji2's by swapping global attention for local attention caused a large performance drop — the gain is the receptive field, not merely the parameter count.
  • The custom relative positional encodings noticeably outperformed both the standard NLP relative bases and absolute positional encodings.
  • More parameters helped, consistent with contemporaneous NLP scaling results.

Performance

Gene expression. Mean correlation for CAGE at the TSS of human protein-coding genes rose from 0.81 (Basenji2) to 0.85 — twice the size of the Basenji1→Basenji2 improvement, and closing one third of the gap to the experimental-replicate ceiling of 0.94.[1] Against ExPecto, Spearman r improved from 0.812 to 0.850 across genes and from 0.368 to 0.451 across tissues.[1] Gains were consistent across all four assay types, largest for CAGE.

Enhancer–gene linking. Evaluated against two K562 CRISPRi screens covering >10,000 candidate enhancers, Enformer contribution scores beat Basenji2 and random across almost all distances — and were comparable to or better than the ABC score, which consumes Hi-C interaction frequency and H3K27ac as input.[1] Enformer uses only DNA sequence and was never trained to locate enhancers. See Activity-by-Contact (ABC) Model.

Insulators, learned implicitly. Averaged attention matrices over 1,500 sequences centred on TAD boundaries show significantly more attention to the boundary and significantly less attention across it, matching the biology of reduced inter-TAD interaction. CTCF was a key motif driving DNase and CAGE predictions, carrying both positive and negative contribution scores.[1]

eQTLs. Two evaluations:[1]

  • Signed LD profile regression against GTEx: the maximum SLDP Z-score increased for 379 of 648 (59.4%) CAGE datasets relative to Basenji2, with 228 increasing by more than one s.d. against 46 decreasing. Mean maximum Z rose 6.3 → 6.9.
  • Fine-mapped variant classification (SuSiE causal probability > 0.9 versus < 0.01, eGene-matched, random forest over the 5,313-feature REF-minus-ALT difference vector): Enformer beat Basenji2 in 47 of 48 GTEx tissues, mean auROC 0.729 → 0.747, consistently across every TSS-distance bin — so the improvement is not solely from the longer receptive field but also from better parsing of promoters and short-range enhancers. Direction-of-effect prediction also improved.

MPRA (CAGI5). Lasso regression over Enformer features achieved the best average correlation across all loci among seven alternative competition submissions, and significantly beat the CAGI5 winning team (P = 0.002). Using Enformer predictions directly, with no additional training, performed comparably and still beat the other submissions.[1]

Stated advantages and limitations

The authors name three advantages of a sequence-only model:[1]

  1. Signed prediction of activating versus repressive effects, which most alternatives cannot do.
  2. No dependence on nucleotide conservation, so predictions are not restricted to conserved enhancers — a small proportion of all enhancers.
  3. Applicability to arbitrary sequences, enabling synthetic enhancer design.

The limitation they state plainly is the one that still constrains the model class: "we can model and predict only for cell types and assays in the training data and cannot generalize to new cell types or assays."[1] They also flag that CRISPR and MPRA datasets were too small at the time to train on, restricting them to evaluation only.

Availability

Pretrained model on TF-Hub (https://tfhub.dev/deepmind/enformer/1), code under Apache 2.0 at the DeepMind research repository and archived at Zenodo, with layer components also in Calico's Basenji repository.[1] Precomputed variant effect predictions are released for all 1000 Genomes variants at MAF > 0.5% in any population, with the 5,313 features distilled by PCA to 20 variant scores — cutting the release from ~100 GB to under 1 GB for 10 million variants while retaining GTEx fine-mapping auROC of 0.743 versus 0.747 with all features.[1]

Position in the model lineage

Synthesis: Enformer, Borzoi and AlphaGenome form a direct three-step lineage, and each step traded a different constraint. Enformer bought receptive field with attention (20 kb → 100 kb) but kept 128 bp bins and predicted CAGE rather than RNA-seq [1]. Borzoi kept the trunk, added a U-Net decoder to reach 32 bp, and switched the primary target to RNA-seq coverage so that splicing and polyadenylation became readable from the same output [2]. AlphaGenome went to 1 Mb at base-pair resolution with explicit per-modality heads [3]. Reading the three papers together, the recurring lesson is that the binding constraint has been output resolution and target choice at least as much as context length — a framing none of the three states directly. See Sequence-to-Function Genomic Models.

A concrete continuity worth noting: Enformer's ABC-score comparison [1] and AlphaGenome's ENCODE–rE2G comparison [3] are the same experiment five years apart — a sequence-only model matching a method that consumes measured chromatin and contact data. Both models reached parity; AlphaGenome additionally improved the supervised model when used as a feature source [3].

See Also

Citations

[1] Avsec, Ž., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska-Barwinska, A., Taylor, K. R., Assael, Y., Jumper, J., Kohli, P. & Kelley, D. R. (2021). Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods 18:1196–1203. Supports: architecture, receptive field and the 47%→84% enhancer-coverage figure; all ablation results; the 0.81→0.85 CAGE and ExPecto comparisons; CRISPRi enhancer-prioritisation and ABC comparison; TAD-boundary attention analysis; SLDP and fine-mapped eQTL classification figures; CAGI5 results; stated advantages, the cell-type generalisation limitation, and all availability details. Location: Results (all sections); Discussion; Data and Code availability; Figs. 1–4. Source paper: s41592-021-01252-x.pdf

[2] Linder, J. et al. (2025). Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. Nature Genetics 57:949–961. Supports: Borzoi's reuse of the Enformer trunk and its U-Net extension to 32 bp. Location: Results ("RNA-seq model design"). Source paper: s41588-024-02053-6.pdf

[3] Avsec, Ž. et al. (2026). Advancing regulatory variant effect prediction with AlphaGenome. Nature 649:1206–1217. Supports: AlphaGenome's 1 Mb / base-pair design, and the ENCODE–rE2G zero-shot and feature-augmentation results referenced in the lineage note. Location: "Unifying DNA sequence-to-function model"; "Competitive enhancer-gene linking". Source paper: s41586-025-10014-0.pdf