Skip to content

Delphi-2M

Summary

Delphi-2M (Delphi large predictive health inference) is an attention-based generative transformer model designed to learn lifetime health trajectories and predict disease risk across continuous time scales. Based on the GPT-2 architecture, Delphi-2M models patient histories as sequences of diagnostic codes (ICD-10), demographics, lifestyle indicators, and artificial "no-event" padding tokens. The model predicts the rates and timings of more than 1,000 diseases simultaneously, providing generative capabilities to simulate synthetic future health trajectories and estimate long-term disease burdens.

Architecture and Design

Delphi-2M extends the standard GPT-2 decoder-only architecture to accommodate continuous, non-discrete health data through three key modifications: - Continuous Age Encoding: Replaces discrete positional encodings with continuous representations of patient age using sine and cosine basis functions. - Exponential Waiting Time Head: Adds a secondary output head that uses an exponential waiting time model (competing exponentials) to predict the continuous time interval to the next diagnostic event. - Causal Continuous-Time Masking: Modifies standard causal attention masks to additionally mask tokens recorded at the exact same age, preventing information leakage among co-occurring events.

The baseline model utilizes 12 layers, 12 attention heads, and an embedding dimension of 120, totaling approximately 2.2 million parameters. The input vocabulary consists of 1,258 tokens (including 1,257 ICD-10 level-3 diagnostic codes, sex, body mass index, smoking, and alcohol frequency indicators).

Training and Validation

Delphi-2M was developed and validated on large-scale population cohorts: - Training Cohort: Trained on electronic health records (EHR) of 402,799 (80%) participants from the UK Biobank recorded before July 1, 2020. - Validation Cohort: Evaluated on 100,639 (20%) UK Biobank participants, achieving an average age-stratified area under the receiver operating characteristic curve (AUC) of 0.76 for next-diagnosis prediction. - External Validation: Validated with zero parameter changes on 1.93 million Danish nationals from Danish population registries (1978–2018), maintaining an average predictive AUC of 0.67.

Applications and Interpretability

  • Disease Trajectory Simulation: Generates synthetic patient health profiles up to 20 years into the future by iteratively sampling the next disease token and wait time. Delphi-2M models trained exclusively on this synthetic data achieved a predictive AUC of 0.74, indicating high preservation of epidemiological statistics.
  • SHAP-Based Explainability: Employs Shapley Additive Explanations (SHAP) to quantify the impact of historical health events on future risks. SHAP analysis revealed clinical co-morbidity clustering matching ICD-10 chapters and demonstrated that acute events (e.g., myocardial infarction) have short-term mortality impacts, while chronic diseases (e.g., cancers) show sustained multi-year effects.

Contributor

  • Gerstung Lab — develops machine-learning models of disease progression and clinical trajectories.

Citations

  • Shmatko, A., Jung, A. W., Gaurav, K., Brunak, S., Mortensen, L. H., Birney, E., Fitzgerald, T., & Gerstung, M. (2025). Learning the natural history of human disease with generative transformers. Nature, 647, 248–256. DOI: 10.1038/s41586-025-09529-3. Source paper: s41586-025-09529-3.pdf