Review of 3 Published Models for Performance Comparison

Risk score for the prediction of dementia risk in 20 years among middle aged people: a longitudinal, population-based study

This study follows patients for 20 years with an incident dementia rate of 4%. AUC ranged between 0.70-0.80, sensitivity was 0.77, specificity was 0.63, and the negative predictive value (NPV) was 0.98.

Features included age, sex, years of education, systolic blood pressure, BMI, total cholesterol, physical activity level, APOE $\varepsilon$ 4 status, and follow-up time.

External Validation of the eRADAR Risk Score for Detecting Undiagnosed Dementia in Two Real-World Healthcare Systems

This study follows patients for 12-months after a 2-year lookback window to diagnose incident dementia in the following 12 months. Features include EHR data (including vital signs, diagnoses, medications, and utilization in the prior 2 years). eRADAR risk scores were generated to aid in the identification of the 50% of dementia patients that otherwise go undiagnosed.

Metrics at eRADAR dementia risk score cut-off at the 90th percentile:

Early prediction of Alzheimer’s disease and related dementias using real-world electronic health records

Methodology takeaways

This study demonstrated that data-driven modeling/feature selection outperformed knowledge-driven feature selection. This indicates that ML approaches to ADRD diagnosis can aid clinicians in better predictive diagnostics. The researchers tested both gradient boosting trees and logistic regression models on these varying sets of features.

Knowledge-driven features from clinicians includes medical diagnoses of obesity, diabetes, hyperlipidemia, hypertension, heart disease, stroke, depression, anxiety, concussion, sleep disorders, periodontitis, smoking, and alcohol use; medication exposures, including nonsteroidal anti-inflammatory drugs (NSAIDs), statins, anticholinergics, hormone replacement therapies, antihypertensives, benzodiazepines, and proton pump inhibitors; the most recent vital signs and lab test results, including body mass index (BMI), systolic/diastolic blood pressure, total cholesterol, high-density lipoprotein, glucose, and hemoglobin A1C (HbA1c) in the observation period. The authors encoded diagnoses and medication histories as binary variables in the models. Measurements were categorized based on the reference normal range (e.g., abnormally low, normal, or abnormally high).

For the data-driven features, the authors used all variables captured by the EHRs, including demographic and behavioral variables, such as age, gender, race, ethnicity, marital status, and smoking status. They included all discrete diagnoses, all medications, and all procedure codes recorded in patients’ EHRs as categorical features. To address the sparsity of features, they grouped similar features.

We are especially interested in the 3-year prediction window given it offers a direct comparison to the cohort we have developed.

Our Cohort and Models

The cohort we have developed focuses on clinical diagnoses alone, as medications and procedures were not included in the initial scope of this project. They may be worth considering in future research by the Lary Lab. We also limited our feature set to diagnoses closely related to the measurements initially identified by the Lary Lab in addition to a few extra diagnoses that were significant in the chi-square test. It may be worthwhile to expand the scope of diagnoses included in this cohort. For feasibility and proof of concept, we limited the number of diagnoses given the large number of patients we have, time constraints, and available computational power.

Feature selection is one area of exploration we did not spend as much time on and would recommend future groups investigate.

Model Performance Summary

Confusion Matrices

Best Hyperparameters

Currently, this model generally underperforms what is available in the literature, mainly due to the data that was available. That being said, we believe that more time spent on feature selection and finetuning the hyperparameters of these models could match or improve upon what is available in the literature.

Our NPV and Sensitivity were our closest metrics, while our AUC is with the bounds of some of these papers, abiet on the lower side.

ROC-AUC Curves

Precision-Recall Curves

Modeling without 81 year olds

Removing individuals with the 1937 birth year imputation generally worsened model performance, so we will be leaving them in for now. This problem would benefit from further exploration going forward.