DLKcat Generalization Failure and Train/Test Leakage
Summary¶
Kroll & Lercher (2024) show that DLKcat's widely cited performance largely reflects data leakage rather than genuine generalization: 67.9% of DLKcat's nominal test-set enzymes were also present in its training data, and a further 23.3% had ≥99% sequence identity to a training enzyme. Once this leakage is accounted for, DLKcat's accuracy collapses — for enzymes with less than 60% maximal sequence identity to the training set its R² becomes negative (worse than predicting the mean), and for truly held-out mutation-effect predictions its R² is -0.18 to -0.21, directly contradicting DLKcat's own headline claim of r=0.94 for mutation-effect prediction.
Method¶
The authors re-examined DLKcat's published train/test split and the "well-studied enzyme mutant" panel DLKcat's original paper used to support its mutation-sensitivity claims (r=0.94). They (1) quantified sequence-identity overlap between DLKcat's nominal training and test enzymes, (2) benchmarked DLKcat's R² against a trivial "geometric mean of the 3 most sequence-similar training enzymes" baseline across bins of decreasing maximal training-set sequence identity, and (3) recomputed DLKcat's mutation-effect accuracy restricted to mutants genuinely absent from the training set.
Findings¶
- Leakage in DLKcat's reported test set: 67.9% of DLKcat's test-set enzymes were literally present in the training data (same enzyme-substrate pair or a duplicate measurement); an additional 23.3% had ≥99% sequence identity to a training enzyme. Only a small remainder was genuinely novel.
- Overall accuracy barely beats a trivial baseline: across the full test set (N=1,687), DLKcat achieves R²=0.445 versus R²=0.420 for a naive baseline that simply averages the kcat of the 3 most sequence-similar training enzymes — indicating DLKcat's deep architecture adds little beyond sequence-similarity lookup.
- Negative R² for dissimilar enzymes: for enzymes with less than 60% maximal sequence identity to any training enzyme, DLKcat's R² is negative — its predictions are worse than simply guessing the training-set mean kcat.
- Mutation-effect claims do not replicate on true held-out data: of the 744 mutant entries in DLKcat's own "well-studied enzyme-substrate pairs" validation panel (the basis of its r=0.94 headline claim), 385 were literally present in the training set. Restricting to the 16 truly unseen mutants (≥99% sequence identity to a training enzyme but not the enzyme itself) gives R²=-0.21 — DLKcat's predictions explain none of the true variance in mutation effects. Across the full held-out mutant panel this figure is R²=-0.18.
- Root cause: DLKcat's data-splitting procedure partitions by enzyme-substrate pair rather than by enzyme-reaction or by sequence-clustered groups, so the same underlying kcat measurement (or a near-duplicate of it) can appear in both the training and test partitions.
- Proposed fixes: cluster enzymes by sequence identity (e.g., CD-HIT) before splitting; group all measurements of the same enzyme into a single split; use enzyme-reaction pairs rather than enzyme-substrate pairs to avoid the same kcat value re-appearing across substrates within a split.
Significance¶
This critique does not claim DLKcat is without value — it demonstrates that headline accuracy metrics for enzyme kinetic parameter predictors can be almost entirely an artifact of evaluation methodology, and that a model's real-world utility (predicting kcat for a genuinely novel enzyme or an untested mutant) can be far below what a leakage-contaminated benchmark suggests. This directly motivated the sequence-identity-stratified and cluster-based evaluation protocols adopted by later tools in the field, including TurNuP (from the same lab) and CatPred.
See Also¶
- DLKcat — the model this critique evaluates.
- Enzyme Kinetic Parameter Prediction — field overview contextualizing this dispute.
- TurNuP — a successor model from the same authors, designed explicitly to generalize to dissimilar enzymes.