Skip to content

GENEB

Summary

GENEB evaluates 40 genomic foundation models on 100 tasks across 13 functional categories using a single protocol: freeze the model, extract embeddings, fit a logistic regression probe, report Matthews Correlation Coefficient.[1] Its finding is a methodological one — aggregate leaderboards are unstable: model rankings vary sharply by task category, scale is only a modest and inconsistent predictor of performance, and architecture and pretraining alignment frequently outweigh parameter count.

Source Status

Preprint: compiled from an arXiv preprint (v3) not yet peer reviewed. confidence: medium.

Protocol

  • Probing: frozen embeddings feed a logistic regression (max_iter = 1000), evaluated in 1-shot, 10-shot and full-data regimes, averaged over five fixed seeds {13, 17, 42, 123, 997}.[1]
  • Metric: Matthews Correlation Coefficient, chosen for robustness to class imbalance. Tasks over 10⁵ sequences are subsampled — an empirical analysis showed MCC stabilises beyond that size.[1]
  • Aggregation: macro-averaged MCC weighting all 13 categories equally; macro and micro rankings correlate at ρ = 0.988.[1]

Freezing the representation is the design decision that makes the comparison controlled — it isolates representation quality from fine-tuning budget and hyperparameter search, which is where most cross-paper comparisons become incommensurable.

Findings

Scale correlates but does not determine. Spearman ρ between log(parameters) and macro-MCC is 0.573 (P < 0.001), rising to 0.694 once the prokaryotic-only outlier EVO-1-131K is excluded.[1] But among 36 in-domain models there are 25 instances where a model at least 5× smaller outperforms a larger one — for example MUTBERT (86M, Transformer-encoder) exceeds ECCDNAMAMBA (537M, Mamba) by +0.110 macro-MCC despite a 6.2-fold size difference.[1] Category-level scaling is significant in 11 of 13 categories but with ρ ranging only 0.346 (DNA methylation) to 0.587 (histone modifications); species classification and chromatin accessibility show no significant scaling.[1]

Architecture matters more than scale on hard categories. Under matched pretraining corpus and tokenization, Transformers beat the evaluated state-space model substantially (GENOMEOCEAN-500M exceeds ECCDNAMAMBA by +0.131 macro-MCC; OMNI-DNA-1B by +0.149), and encoders beat decoders across all six matched pairs.[1] On cross-species regulatory tasks the gaps widen dramatically — +0.355 macro-MCC on virus/phage, +0.305 on mouse enhancers — several-fold larger than the +0.075 aggregate gain between models above 1B and below 200M parameters.[1]

One clean architectural exception. On chromatin accessibility, the Mamba model beats the Transformer (+0.124 macro-MCC), and both SSM-family models show a within-model advantage on this category relative to their own aggregate (+0.200 and +0.179).[1] Multi-species pretraining adds a further +0.076 for this category in 6/6 matched pairs.

Tokenization admits no global ordering. Across 11 matched controlled pairs, the preferred scheme varies with model family and pretraining setting — BPE beats k-mer for one decoder pair (+0.032), BPE and k-mer are comparable within encoders (+0.006 across 5 pairs), and single-nucleotide tokenization beats BPE under matched human pretraining (+0.033, +0.038).[1]

Pretraining scope predicts tier membership. On the 13 highest-variance tasks (cross-model SD > 0.12), multi-species and eukaryotic-gene pretraining capture 32/39 top-3 placements, while human-only pretraining takes 29/39 bottom-3 and only 1/39 top-3.[1] Microbial-only corpora transfer poorly to eukaryotic tasks.

Few-shot reranks the winner in 8 of 13 categories — the model that wins under full supervision is often not the one to deploy with ~10 labelled examples.[1]

A hard frontier remains. 28 of 100 tasks have mean MCC below 0.35, dominated by 4mC methylation prediction (as low as 0.061) and plant lncRNA identification (0.221–0.238). The best model reaches only 0.440 on DNA methylation and 0.508 on plant lncRNA.[1]

Relevance to this vault

Enformer appears in GENEB as a CNN–Transformer hybrid and is named, alongside SPACE, as consistently among the top models for epigenomic-profile tasks (TF binding, regulatory, enhancers), matching the overall top model OMNI-DNA-1B on mouse enhancers.[1] That is an independent, protocol-controlled corroboration of the enhancer-prioritisation capability reported in Enformer's own paper.

Synthesis: GENEB's central claim — that aggregate rankings hide category-level reversals — is the same hazard this vault records for splicing models from a different direction. Pangolin beats models with 100× its receptive field on specific splicing benchmarks [2], and Borzoi and Pangolin's ensemble beats either alone [3]. GENEB generalises the pattern across 40 models and 100 tasks and gives it a protocol. Neither source discusses the other; the parallel is drawn here. See also DLKcat Generalization Failure and Train/Test Leakage for the same failure mode in enzyme kinetics.

Stated limitations

  • Long-range tasks are underrepresented (>10 kb regulatory interactions), so models with explicit long-context architectures (HYENADNA-LARGE-1M, CADUCEUS-PS-131K, EVO-1-131K) are never exercised where their priors would help most.[1] This directly limits GENEB's relevance to the sequence-to-function models that operate at 200 kb–1 Mb.
  • Task quality varies, particularly in the hard regimes where label quality and supervision signal differ across sources.[1]
  • Aggregate rankings are explicitly declared a poor proxy for prokaryotic or viral genomics; users in those domains are told to consult per-task results.[1]

See Also

Citations

[1] Ledneva, D., Nuridinov, M. & Kuznetsov, D. (2026). GENEB: Why Genomic Models Are Hard to Compare. arXiv preprint arXiv:2606.04525v3. Supports: the probing protocol and metric; all scaling, architecture, tokenization and pretraining-corpus results; the high-variance-task tier analysis; few-shot reranking; the hard-frontier task list; Enformer's placement; all stated limitations. Location: Methodology (§3); Aggregate Performance Analysis (§4) including "The Hard Frontier of GENEB" and "Practitioner Recommendations"; Conclusion (§5); Limitations (§6); Figs. 2–6; Table 1. Source paper: 2606.04525v3.pdf

[2] Zeng, T. & Li, Y. I. (2022). Predicting RNA splicing from DNA sequence using Pangolin. Genome Biology 23:103. Supports: the splicing-model counter-example cited in the Synthesis block. Location: Main text. Source paper: s13059-022-02664-4.pdf

[3] Linder, J. et al. (2025). Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. Nature Genetics 57:949–961. Supports: the Borzoi/Pangolin ensemble gain cited in the Synthesis block. Location: Results ("Functional splicing variant interpretation"). Source paper: s41588-024-02053-6.pdf