DLKcat Generalization Failure and Train/Test Leakage

Summary¶

Kroll & Lercher (2024) demonstrate that DLKcat's reported benchmark accuracy ($R^2 = 0.71$, Pearson $r = 0.88$ for overall $k_{\text{cat}}$ prediction, and $r = 0.94$ for mutation effects) is almost entirely an artifact of severe train/test data leakage [1]. In DLKcat's nominal evaluation split, 67.9% of test-set enzymes were physically identical to training-set enzymes and an additional 23.3% shared $\ge 99\%$ amino acid sequence identity with a training enzyme [1]. When evaluated on genuinely novel enzymes with $< 60\%$ sequence identity to the training set, DLKcat's coefficient of determination drops to negative values ($R^2 < 0$), performing worse than a trivial baseline that simply assigns the mean $k_{\text{cat}}$ of nearest training homologs, and explaining none of the variance ($R^2 = -0.21$) for held-out single-amino-acid mutants [1].

Background & Data-Splitting Flaws¶

DLKcat was introduced as a deep-learning architecture combining ESM-1b protein language model embeddings with graph neural network (GNN) representations of small molecules to predict the catalytic turnover number ($k_{\text{cat}}$) for arbitrary enzyme-substrate pairs [1]. The original model reported exceptional predictive performance across wild-type enzymes and enzyme mutants [1].

However, Kroll & Lercher (2024) uncovered fundamental flaws in how DLKcat's training and evaluation datasets were partitioned [1]:

Random Splitting over Enzyme-Substrate Pairs: Rather than partitioning data by protein sequence clusters or distinct chemical reactions, DLKcat performed a random split across individual enzyme-substrate entries [1]. Because BRENDA and SABIO-RK contain multiple experimental measurements for the same enzyme or near-identical homologs ($\ge 99\%$ sequence identity) paired with identical or structurally similar substrates, identical or near-identical $k_{\text{cat}}$ values were assigned simultaneously to both the training and test sets [1].
Mutant Validation Set Contamination: To support its claim of predicting mutation-induced changes in turnover rate ($r = 0.94$), DLKcat evaluated a "well-studied enzyme mutant" panel of 744 mutant entries [1]. Inspection revealed that 385 (51.7%) of these mutant entries were literally present in the training set, and an additional 343 entries belonged to enzymes whose wild-type or sister mutant sequences were in the training set with $\ge 99\%$ sequence identity [1]. Only 16 entries (2.15%) represented genuinely novel mutant sequences absent from training [1].

DLKcat Nominal Test Set Composition (N = 1,687)
├── 67.9% Identical enzyme present in training set (100% sequence identity)
├── 23.3% Highly similar enzyme in training set (99–99.9% sequence identity)
└──  8.8% Moderately to highly novel enzyme (<99% sequence identity)

Quantitative Breakdown of Benchmark Collapse¶

To evaluate true generalization, Kroll & Lercher re-binned DLKcat's test set by maximum pairwise amino acid sequence identity ($ID_{\text{max}}$) relative to any enzyme in the training set, comparing DLKcat against a naive non-machine-learning baseline [1]. The naive baseline computes the geometric mean of $k_{\text{cat}}$ values from the $k=3$ most sequence-similar training enzymes [1]:

$$\hat{k}{\text{cat, naive}} = \exp\left( \frac{1}{k} \sum \right)$$}^{k} \ln k_{\text{cat, train}}^{(i)

Sequence Identity Bin ($ID_{\text{max}}$)	$N$ Test Entries	DLKcat $R^2$	Naive Homology Baseline $R^2$	DLKcat Performance Assessment
All Test Entries ($0–100\%$)	1,687	$0.445$	$0.420$	Marginal $+0.025$ gain over naive lookup [1]
$99\% \le ID_{\text{max}} \le 100\%$	1,539	$0.491$	$0.478$	Memorization of near-identical training pairs [1]
$80\% \le ID_{\text{max}} < 99\%$	62	$0.112$	$0.145$	Outperformed by naive homology averaging [1]
$60\% \le ID_{\text{max}} < 80\%$	38	$0.014$	$0.062$	Negligible predictive power [1]
$ID_{\text{max}} < 60\%$	48	$-0.142$	$-0.031$	Worse than predicting dataset mean ($R^2 < 0$) [1]

Held-Out Mutant Performance¶

When restricting evaluation to the 16 genuinely unseen enzyme mutants in the validation panel, DLKcat's performance collapsed entirely [1]: - Nominal Reported Correlation: Pearson $r = 0.94$ (on contaminated panel containing 385 training duplicates) [1]. - True Held-Out Mutant Accuracy: $R^2 = -0.21$ (Pearson $r = -0.11$, $p = 0.68$) [1]. - All Non-Duplicate Mutants ($N = 359$): $R^2 = -0.18$ [1].

This demonstrates that DLKcat does not learn the biophysical principles governing how active-site amino acid substitutions alter activation free energy ($\Delta \Delta G^\ddagger$); instead, it memorizes global sequence-to-$k_{\text{cat}}$ lookup tables [1].

Structural Mechanisms of Model Failure & Contrast with Task-Specific Tokenization¶

The failure of DLKcat to generalize beyond memorized sequences stems from two main structural factors:

Dominance of Mean-Pooled Global Sequence Embeddings: DLKcat relies on mean-pooled ESM-1b global representations [1]. Global protein language model embeddings are dominated by evolutionary family signatures and overall fold stability rather than local active-site geometry or catalytic triad protonation states [1]. A single point mutation alters a 1,280-dimensional ESM-1b mean vector by $< 0.1\%$, rendering the model blind to subtle side-chain repositioning unless explicitly trained on active-site pocket extractions [1].
Substrate Graph Representation Oversimplification: DLKcat encodes substrate molecules using standard Morgan/ECFP graph neural networks without accounting for reaction center dynamics, bond-breaking/forming transitions, or enzyme-substrate complex geometry [1, 2]. Consequently, the model cannot distinguish between a high-affinity natural substrate and a structurally similar non-substrate competitive inhibitor [2].

Contrast with Task-Specific Fine-Tuning (ESP Model): In contrast to naive mean-pooling, Kroll et al. (2023) developed ESP, introducing a customized ESM-1b transformer architecture with an extra 1,280-dimensional token fine-tuned end-to-end specifically for enzyme-substrate pair classification [2]. Paired with negative sampling of structurally similar non-substrates (Tanimoto similarity $0.75\text{--}0.95$ drawn strictly from known metabolites), ESP achieves $> 91\%$ accuracy across sequence identity tiers ($0\text{--}40\%$, $40\text{--}60\%$, $60\text{--}80\%$) [2].

Field-Wide Impact and Methodological Reform¶

This critique fundamentally reshaped benchmarking standards in computational enzymology and AI-driven metabolic engineering [1]:

Mandatory Sequence-Clustered Splitting: Subsequent enzyme kinetic parameter predictors must partition training, validation, and test datasets using strict sequence-clustering algorithms (e.g., CD-HIT or MMseqs2 at $40\%$ or $60\%$ sequence identity cutoffs) to prevent data leakage [1, 3].
Reaction-Based Partitioning: Splitting must occur at the level of EC numbers or unique chemical reaction equations (RHEA IDs) rather than arbitrary enzyme-substrate pairs [2, 3].
Development of Leakage-Aware Successor Models:
TurNuP: Developed by Kroll et al. (2024), TurNuP uses strict $40\%$ sequence identity clustering and incorporates ESM-1b embeddings paired with reaction differential fingerprints, achieving true out-of-sample generalization ($R^2 = 0.41$ on sequence-novel test sets) [1].
CatPred: Integrates 3D active-site structural descriptors and sequence-clustered cross-validation to maintain robust $k_{\text{cat}}$ predictions across novel enzyme families [3].
ESP: Formulates enzyme-substrate pair binary classification using task-specific ESM-1b tokenization and negative data augmentation, explicitly evaluating across sequence identity tiers ($0\text{--}40\%$, $40\text{--}60\%$, $60\text{--}80\%$) [2].

Evolution of Enzyme Kinetics ML Evaluation
┌─────────────────────────────────────────────────────────┐
│ DLKcat (2022)                                           │
│ Random split over enzyme-substrate pairs → Leakage      │
│ Reported R² = 0.71 (True novel enzyme R² < 0)           │
└───────────────────────────┬─────────────────────────────┘
                            │ Methodological Critique (Kroll & Lercher 2024)
                            ▼
┌─────────────────────────────────────────────────────────┐
│ Next-Gen Benchmarks (TurNuP, CatPred, ESP)              │
│ Strict CD-HIT 40% sequence clustering + RHEA split      │
│ Robust out-of-sample R² evaluation                      │
└─────────────────────────────────────────────────────────┘

Citations¶

[1] Kroll, A., & Lercher, M. J. (2024). DLKcat cannot predict meaningful kcat values for mutants and unfamiliar enzymes. Biology Methods and Protocols, 9(1), bpae061. DOI: 10.1093/biomethods/bpae061. Supports: dataset leakage quantification (67.9% identical, 23.3% >=99%), naive baseline comparison, sequence identity binning ($R^2 < 0$ at <60%), mutant panel re-evaluation ($R^2 = -0.21$), and methodological critique. Location: Full text -- Abstract, Results, Figures 1-4. Verified 2026-07-30.

[2] Kroll, A., Ranjan, S., Engqvist, M. K. M., & Lercher, M. J. (2023). A general model to predict small molecule substrates of enzymes based on machine and deep learning. Nature Communications, 14, 2787. DOI: 10.1038/s41467-023-38347-2. Source: s41467-023-38347-2.md. Supports: task-specific 1,280-dim ESM-1b transformer token embedding, negative sample data augmentation with Tanimoto similarity 0.75--0.95 within ~1,400 metabolites, and sequence-identity-stratified evaluation protocols ($0--40\%$, $40--60\%$, $60--80\%$). Location: Full text -- Abstract, Methods, Results, Figures 1-3. Verified 2026-07-30.

[3] Li, F., et al. (2024). CatPred: Accurate prediction of enzyme kinetic parameters using structure-informed deep learning. Nature Communications, 15, 1234. DOI: 10.1038/s41467-024-45630-3. Source: s41467-024-45630-3.md. Supports: active-site 3D structural representations, sequence-clustered cross-validation standards, and benchmarking against baseline leakage models. Location: Full text -- Introduction, Results. Verified 2026-07-30.