Dataset Information

Internal and External Validation of a Machine Learning Risk Score for Acute Kidney Injury.

ABSTRACT:

Importance

Acute kidney injury (AKI) is associated with increased morbidity and mortality in hospitalized patients. Current methods to identify patients at high risk of AKI are limited, and few prediction models have been externally validated.

Objective

To internally and externally validate a machine learning risk score to detect AKI in hospitalized patients.

Design, setting, and participants

This diagnostic study included 495 971 adult hospital admissions at the University of Chicago (UC) from 2008 to 2016 (n = 48 463), at Loyola University Medical Center (LUMC) from 2007 to 2017 (n = 200 613), and at NorthShore University Health System (NUS) from 2006 to 2016 (n = 246 895) with serum creatinine (SCr) measurements. Patients with an SCr concentration at admission greater than 3.0 mg/dL, with a prior diagnostic code for chronic kidney disease stage 4 or higher, or who received kidney replacement therapy within 48 hours of admission were excluded. A simplified version of a previously published gradient boosted machine AKI prediction algorithm was used; it was validated internally among patients at UC and externally among patients at NUS and LUMC.

Main outcomes and measures

Prediction of Kidney Disease Improving Global Outcomes SCr-defined stage 2 AKI within a 48-hour interval was the primary outcome. Discrimination was assessed by the area under the receiver operating characteristic curve (AUC).

Results

The study included 495 971 adult admissions (mean [SD] age, 63 [18] years; 87 689 [17.7%] African American; and 266 866 [53.8%] women) across 3 health systems. The development of stage 2 or higher AKI occurred in 15 664 of 48 463 patients (3.4%) in the UC cohort, 5711 of 200 613 (2.8%) in the LUMC cohort, and 3499 of 246 895 (1.4%) in the NUS cohort. In the UC cohort, 332 patients (0.7%) required kidney replacement therapy compared with 672 patients (0.3%) in the LUMC cohort and 440 patients (0.2%) in the NUS cohort. The AUCs for predicting at least stage 2 AKI in the next 48 hours were 0.86 (95% CI, 0.86-0.86) in the UC cohort, 0.85 (95% CI, 0.84-0.85) in the LUMC cohort, and 0.86 (95% CI, 0.86-0.86) in the NUS cohort. The AUCs for receipt of kidney replacement therapy within 48 hours were 0.96 (95% CI, 0.96-0.96) in the UC cohort, 0.95 (95% CI, 0.94-0.95) in the LUMC cohort, and 0.95 (95% CI, 0.94-0.95) in the NUS cohort. In time-to-event analysis, a probability cutoff of at least 0.057 predicted the onset of stage 2 AKI a median (IQR) of 27 (6.5-93) hours before the eventual doubling in SCr concentrations in the UC cohort, 34.5 (19-85) hours in the NUS cohort, and 39 (19-108) hours in the LUMC cohort.

Conclusions and relevance

In this study, the machine learning algorithm demonstrated excellent discrimination in both internal and external validation, supporting its generalizability and potential as a clinical decision support tool to improve AKI detection and outcomes.

SUBMITTER: Churpek MM

PROVIDER: S-EPMC7420241 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectiveTo develop and externally validate a prediction model for severe cisplatin associated acute kidney injury (CP-AKI).DesignMulticenter cohort study.SettingSix geographically diverse major academic cancer centers across the US.ParticipantsAdults (≥18 years) receiving their first dose of intravenous cisplatin, 2006-22.Main outcome measuresThe primary outcome was CP-AKI, defined as a twofold or greater increase in serum creatinine or kidney replacement therapy within 14 days of a first dose of intravenous cisplatin. Independent predictors of CP-AKI were identified using a multivariable logistic regression model, which was developed in a derivation cohort and tested in an external validation cohort. For the primary model, continuous variables were examined using restricted cubic splines. A simple risk model was also generated by converting the odds ratios from the primary model into risk points. Finally, a multivariable Cox model was used to examine the association between severity of CP-AKI and 90 day survival.ResultsA total of 24 717 adults were included, with 11 766 in the derivation cohort (median age 59 (interquartile range (IQR) 50-67)) and 12 951 in the validation cohort (median age 60 (IQR 50-67)). The incidence of CP-AKI was 5.2% (608/11 766) in the derivation cohort and 3.3% (421/12 951) in the validation cohort. Each of the following factors were independently associated with CP-AKI in the derivation cohort: age, hypertension, diabetes mellitus, serum creatinine level, hemoglobin level, white blood cell count, platelet count, serum albumin level, serum magnesium level, and cisplatin dose. A simple risk score consisting of nine covariates was shown to predict a higher risk of CP-AKI in a monotonic fashion in both the derivation cohort and the validation cohort. Compared with patients in the lowest risk category, those in the highest risk category showed a 24.00-fold (95% confidence interval (CI) 13.49-fold to 42.78-fold) higher odds of CP-AKI in the derivation cohort and a 17.87-fold (10.56-fold to 29.60-fold) higher odds in the validation cohort. The primary model had a C statistic of 0.75 and showed better discrimination for CP-AKI than previously published models, the C statistics for which ranged from 0.60 to 0.68 (DeLong P<0.001 for each comparison). Greater severity of CP-AKI was monotonically associated with shorter 90 day survival (adjusted hazard ratio 4.63 (95% CI 3.56 to 6.02) for stage 3 CP-AKI versus no CP-AKI).ConclusionThis study found that a simple risk score based on readily available variables from patients receiving intravenous cisplatin could predict the risk of severe CP-AKI, the occurrence of which is strongly associated with death.

Project description:ObjectiveWe aimed to develop and validate a predictive machine learning (ML) model for cardiac surgery associated with acute kidney injury (CSA-AKI) based on a multicenter randomized control trial (RCT) and a Medical Information Mart for Intensive Care-IV (MIMIC-IV) dataset.MethodsThis was a subanalysis from a completed RCT approved by the Ethics Committee of Fuwai Hospital in Beijing, China (NCT03782350). Data from Fuwai Hospital were randomly assigned, with 80% for the training dataset and 20% for the testing dataset. The data from three other centers were used for the external validation dataset. Furthermore, the MIMIC-IV dataset was also utilized to validate the performance of the predictive model. The area under the receiver operating characteristic curve (ROC-AUC), the precision-recall curve (PR-AUC), and the calibration brier score were applied to evaluate the performance of the traditional logistic regression (LR) and eleven ML algorithms. Additionally, the Shapley Additive Explanations (SHAP) interpreter was used to explain the potential risk factors for CSA-AKI.ResultA total of 6495 eligible patients undergoing cardiopulmonary bypass (CPB) were eventually included in this study, 2416 of whom were from Fuwai Hospital (Beijing), for model development, 562 from three other cardiac centers in China, and 3517 from the MIMICIV dataset, were used, respectively, for external validation. The CatBoostClassifier algorithms outperformed other models, with excellent discrimination and calibration performance for the development, as well as the MIMIC-IV, datasets. In addition, the CatBoostClassifier achieved ROC-AUCs of 0.85, 0.67, and 0.77 and brier scores of 0.14, 0.19, and 0.16 in the testing, external, and MIMIC-IV datasets, respectively. Moreover, the utmost important risk factor, the N-terminal brain sodium peptide (NT-proBNP), was confirmed by the LASSO method in the feature section process. Notably, the SHAP explainer identified that the preoperative blood urea nitrogen level, prothrombin time, serum creatinine level, total bilirubin level, and age were positively correlated with CSA-AKI; preoperative platelets level, systolic and diastolic blood pressure, albumin level, and body weight were negatively associated with CSA-AKI.ConclusionsThe CatBoostClassifier algorithms outperformed other ML models in the discrimination and calibration of CSA-AKI prediction cardiac surgery with CPB, based on a multicenter RCT and MIMIC-IV dataset. Moreover, the preoperative NT-proBNP level was confirmed to be strongly related to CSA-AKI.

Project description:BACKGROUND:Community-acquired acute kidney injury (CA-AKI)-associated hospitalizations impose significant health care needs and contribute to in-hospital mortality. However, most risk prediction models developed to date have focused on AKI in a specific group of patients during hospitalization, and there is limited knowledge on the baseline risk in the general population for preventing CA-AKI-associated hospitalization. OBJECTIVE:To gain further insight into risk exploration, the aim of this study was to develop, validate, and establish a scoring system to facilitate health professionals in enabling early recognition and intervention of CA-AKI to prevent permanent kidney damage using different machine-learning techniques. METHODS:A nested case-control study design was employed using electronic health records derived from a group of Chang Gung Memorial Hospitals in Taiwan from 2010 to 2017 to identify 234,867 adults with at least two measures of serum creatinine at hospital admission. Patients were classified into a derivation cohort (2010-2016) and a temporal validation cohort (2017). Patients with the first episode of CA-AKI at hospital admission were classified into the case group and those without CA-AKI were classified in the control group. A total of 47 potential candidate variables, including age, gender, prior use of nephrotoxic medications, Charlson comorbid conditions, commonly measured laboratory results, and recent use of health services, were tested to develop a CA-AKI hospitalization risk model. Permutation-based selection with both the extreme gradient boost (XGBoost) and least absolute shrinkage and selection operator (LASSO) algorithms was performed to determine the top 10 important features for scoring function development. RESULTS:The discriminative ability of the risk model was assessed by the area under the receiver operating characteristic curve (AUC), and the predictive CA-AKI risk model derived by the logistic regression algorithm achieved an AUC of 0.767 (95% CI 0.764-0.770) on derivation and 0.761 on validation for any stage of AKI, with positive and negative predictive values of 19.2% and 96.1%, respectively. The risk model for prediction of CA-AKI stages 2 and 3 had an AUC value of 0.818 for the validation cohort with positive and negative predictive values of 13.3% and 98.4%, respectively. These metrics were evaluated at a cut-off value of 7.993, which was determined as the threshold to discriminate the risk of AKI. CONCLUSIONS:A machine learning-generated risk score model can identify patients at risk of developing CA-AKI-related hospitalization through a routine care data-driven approach. The validated multivariate risk assessment tool could help clinicians to stratify patients in primary care, and to provide monitoring and early intervention for preventing AKI while improving the quality of AKI care in the general population.

Project description:ImportanceSevere acute kidney injury (AKI) is a serious postoperative complication. A tool for predicting the risk of AKI requiring kidney replacement therapy (KRT) after major noncardiac surgery might assist with patient counseling and targeted use of measures to reduce this risk.ObjectiveTo derive and validate a predictive model for AKI requiring KRT after major noncardiac surgery.Design, setting, and participantsIn this prognostic study, 5 risk prediction models were derived and internally validated in a population-based cohort of adults without preexisting kidney failure who underwent noncardiac surgery in Alberta, Canada, between January 1, 2004, and December 31, 2013. The best performing model and corresponding risk index were externally validated in a population-based cohort of adults without preexisting kidney failure who underwent noncardiac surgery in Ontario, Canada, between January 1, 2007, and December 31, 2017. Data analysis was conducted from September 1, 2019, to May 31, 2021.ExposuresDemographic characteristics, surgery type, laboratory measures, and comorbidities before surgery.Main outcomes and measuresAcute kidney injury requiring KRT within 14 days after surgery. Discrimination was assessed using the C statistic; calibration was assessed using calibration intercept and slope. Logistic recalibration was used to optimize model calibration in the external validation cohort.ResultsThe derivation cohort included 92 114 patients (52.2% female; mean [SD] age, 62.3 [18.0] years), and the external validation cohort included 709 086 patients (50.8% female; mean [SD] age, 61.0 [16.0] years). A total of 529 patients (0.6%) developed postoperative AKI requiring KRT in the derivation cohort, and 2956 (0.4%) developed postoperative AKI requiring KRT in the external validation cohort. The following factors were consistently associated with the risk of AKI requiring KRT: younger age (40-69 years: odds ratio [OR], 2.07 [95% CI, 1.69-2.53]; <40 years: OR, 3.73 [95% CI, 2.61-5.33]), male sex (OR, 1.55; 95% CI, 1.28-1.87), surgery type (colorectal: OR, 4.86 [95% CI, 3.28-7.18]; liver or pancreatic: OR, 6.46 [95% CI, 3.85-10.83]; other abdominal: OR, 2.19 [95% CI, 1.66-2.89]; abdominal aortic aneurysm repair: OR, 19.34 [95% CI, 14.31-26.14]; other vascular: OR, 7.30 [95% CI, 5.48-9.73]; thoracic: OR, 3.41 [95% CI, 2.07-5.59]), lower estimated glomerular filtration rate (OR, 0.97; 95% CI, 0.97-0.97 per 1 mL/min/1.73 m2 increase), lower hemoglobin concentration (OR, 0.99; 95% CI, 0.98-0.99 per 0.1 g/dL increase), albuminuria (mild: OR, 1.88 [95% CI, 1.52-2.33]; heavy: OR, 3.74 [95% CI, 2.98-4.69]), history of myocardial infarction (OR, 1.63; 95% CI, 1.32-2.03), and liver disease (mild: OR, 2.32 [95% CI, 1.66-3.24]; moderate or severe: OR, 4.96 [95% CI, 3.58-6.85]). In external validation, a final model including these variables showed excellent discrimination (C statistic, 0.95; 95% CI, 0.95-0.96), with sensitivity of 21.2%, specificity of 99.9%, positive predictive value of 38.1%, and negative predictive value of 99.7% at a predicted risk threshold of 10% or greater.Conclusions and relevanceThe findings suggest that this risk model can predict AKI requiring KRT after noncardiac surgery using routine preoperative data. The model may be feasible for implementation in clinical perioperative risk stratification for severe AKI.