Dataset Information

Internal and External Validation of a Machine Learning Risk Score for Acute Kidney Injury.

ABSTRACT:

Importance

Acute kidney injury (AKI) is associated with increased morbidity and mortality in hospitalized patients. Current methods to identify patients at high risk of AKI are limited, and few prediction models have been externally validated.

Objective

To internally and externally validate a machine learning risk score to detect AKI in hospitalized patients.

Design, setting, and participants

This diagnostic study included 495 971 adult hospital admissions at the University of Chicago (UC) from 2008 to 2016 (n = 48 463), at Loyola University Medical Center (LUMC) from 2007 to 2017 (n = 200 613), and at NorthShore University Health System (NUS) from 2006 to 2016 (n = 246 895) with serum creatinine (SCr) measurements. Patients with an SCr concentration at admission greater than 3.0 mg/dL, with a prior diagnostic code for chronic kidney disease stage 4 or higher, or who received kidney replacement therapy within 48 hours of admission were excluded. A simplified version of a previously published gradient boosted machine AKI prediction algorithm was used; it was validated internally among patients at UC and externally among patients at NUS and LUMC.

Main outcomes and measures

Prediction of Kidney Disease Improving Global Outcomes SCr-defined stage 2 AKI within a 48-hour interval was the primary outcome. Discrimination was assessed by the area under the receiver operating characteristic curve (AUC).

Results

The study included 495 971 adult admissions (mean [SD] age, 63 [18] years; 87 689 [17.7%] African American; and 266 866 [53.8%] women) across 3 health systems. The development of stage 2 or higher AKI occurred in 15 664 of 48 463 patients (3.4%) in the UC cohort, 5711 of 200 613 (2.8%) in the LUMC cohort, and 3499 of 246 895 (1.4%) in the NUS cohort. In the UC cohort, 332 patients (0.7%) required kidney replacement therapy compared with 672 patients (0.3%) in the LUMC cohort and 440 patients (0.2%) in the NUS cohort. The AUCs for predicting at least stage 2 AKI in the next 48 hours were 0.86 (95% CI, 0.86-0.86) in the UC cohort, 0.85 (95% CI, 0.84-0.85) in the LUMC cohort, and 0.86 (95% CI, 0.86-0.86) in the NUS cohort. The AUCs for receipt of kidney replacement therapy within 48 hours were 0.96 (95% CI, 0.96-0.96) in the UC cohort, 0.95 (95% CI, 0.94-0.95) in the LUMC cohort, and 0.95 (95% CI, 0.94-0.95) in the NUS cohort. In time-to-event analysis, a probability cutoff of at least 0.057 predicted the onset of stage 2 AKI a median (IQR) of 27 (6.5-93) hours before the eventual doubling in SCr concentrations in the UC cohort, 34.5 (19-85) hours in the NUS cohort, and 39 (19-108) hours in the LUMC cohort.

Conclusions and relevance

In this study, the machine learning algorithm demonstrated excellent discrimination in both internal and external validation, supporting its generalizability and potential as a clinical decision support tool to improve AKI detection and outcomes.

SUBMITTER: Churpek MM

PROVIDER: S-EPMC7420241 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:Community-acquired acute kidney injury (CA-AKI)-associated hospitalizations impose significant health care needs and contribute to in-hospital mortality. However, most risk prediction models developed to date have focused on AKI in a specific group of patients during hospitalization, and there is limited knowledge on the baseline risk in the general population for preventing CA-AKI-associated hospitalization. OBJECTIVE:To gain further insight into risk exploration, the aim of this study was to develop, validate, and establish a scoring system to facilitate health professionals in enabling early recognition and intervention of CA-AKI to prevent permanent kidney damage using different machine-learning techniques. METHODS:A nested case-control study design was employed using electronic health records derived from a group of Chang Gung Memorial Hospitals in Taiwan from 2010 to 2017 to identify 234,867 adults with at least two measures of serum creatinine at hospital admission. Patients were classified into a derivation cohort (2010-2016) and a temporal validation cohort (2017). Patients with the first episode of CA-AKI at hospital admission were classified into the case group and those without CA-AKI were classified in the control group. A total of 47 potential candidate variables, including age, gender, prior use of nephrotoxic medications, Charlson comorbid conditions, commonly measured laboratory results, and recent use of health services, were tested to develop a CA-AKI hospitalization risk model. Permutation-based selection with both the extreme gradient boost (XGBoost) and least absolute shrinkage and selection operator (LASSO) algorithms was performed to determine the top 10 important features for scoring function development. RESULTS:The discriminative ability of the risk model was assessed by the area under the receiver operating characteristic curve (AUC), and the predictive CA-AKI risk model derived by the logistic regression algorithm achieved an AUC of 0.767 (95% CI 0.764-0.770) on derivation and 0.761 on validation for any stage of AKI, with positive and negative predictive values of 19.2% and 96.1%, respectively. The risk model for prediction of CA-AKI stages 2 and 3 had an AUC value of 0.818 for the validation cohort with positive and negative predictive values of 13.3% and 98.4%, respectively. These metrics were evaluated at a cut-off value of 7.993, which was determined as the threshold to discriminate the risk of AKI. CONCLUSIONS:A machine learning-generated risk score model can identify patients at risk of developing CA-AKI-related hospitalization through a routine care data-driven approach. The validated multivariate risk assessment tool could help clinicians to stratify patients in primary care, and to provide monitoring and early intervention for preventing AKI while improving the quality of AKI care in the general population.

Project description:ImportanceSevere acute kidney injury (AKI) is a serious postoperative complication. A tool for predicting the risk of AKI requiring kidney replacement therapy (KRT) after major noncardiac surgery might assist with patient counseling and targeted use of measures to reduce this risk.ObjectiveTo derive and validate a predictive model for AKI requiring KRT after major noncardiac surgery.Design, setting, and participantsIn this prognostic study, 5 risk prediction models were derived and internally validated in a population-based cohort of adults without preexisting kidney failure who underwent noncardiac surgery in Alberta, Canada, between January 1, 2004, and December 31, 2013. The best performing model and corresponding risk index were externally validated in a population-based cohort of adults without preexisting kidney failure who underwent noncardiac surgery in Ontario, Canada, between January 1, 2007, and December 31, 2017. Data analysis was conducted from September 1, 2019, to May 31, 2021.ExposuresDemographic characteristics, surgery type, laboratory measures, and comorbidities before surgery.Main outcomes and measuresAcute kidney injury requiring KRT within 14 days after surgery. Discrimination was assessed using the C statistic; calibration was assessed using calibration intercept and slope. Logistic recalibration was used to optimize model calibration in the external validation cohort.ResultsThe derivation cohort included 92 114 patients (52.2% female; mean [SD] age, 62.3 [18.0] years), and the external validation cohort included 709 086 patients (50.8% female; mean [SD] age, 61.0 [16.0] years). A total of 529 patients (0.6%) developed postoperative AKI requiring KRT in the derivation cohort, and 2956 (0.4%) developed postoperative AKI requiring KRT in the external validation cohort. The following factors were consistently associated with the risk of AKI requiring KRT: younger age (40-69 years: odds ratio [OR], 2.07 [95% CI, 1.69-2.53]; <40 years: OR, 3.73 [95% CI, 2.61-5.33]), male sex (OR, 1.55; 95% CI, 1.28-1.87), surgery type (colorectal: OR, 4.86 [95% CI, 3.28-7.18]; liver or pancreatic: OR, 6.46 [95% CI, 3.85-10.83]; other abdominal: OR, 2.19 [95% CI, 1.66-2.89]; abdominal aortic aneurysm repair: OR, 19.34 [95% CI, 14.31-26.14]; other vascular: OR, 7.30 [95% CI, 5.48-9.73]; thoracic: OR, 3.41 [95% CI, 2.07-5.59]), lower estimated glomerular filtration rate (OR, 0.97; 95% CI, 0.97-0.97 per 1 mL/min/1.73 m2 increase), lower hemoglobin concentration (OR, 0.99; 95% CI, 0.98-0.99 per 0.1 g/dL increase), albuminuria (mild: OR, 1.88 [95% CI, 1.52-2.33]; heavy: OR, 3.74 [95% CI, 2.98-4.69]), history of myocardial infarction (OR, 1.63; 95% CI, 1.32-2.03), and liver disease (mild: OR, 2.32 [95% CI, 1.66-3.24]; moderate or severe: OR, 4.96 [95% CI, 3.58-6.85]). In external validation, a final model including these variables showed excellent discrimination (C statistic, 0.95; 95% CI, 0.95-0.96), with sensitivity of 21.2%, specificity of 99.9%, positive predictive value of 38.1%, and negative predictive value of 99.7% at a predicted risk threshold of 10% or greater.Conclusions and relevanceThe findings suggest that this risk model can predict AKI requiring KRT after noncardiac surgery using routine preoperative data. The model may be feasible for implementation in clinical perioperative risk stratification for severe AKI.

Project description:ImportanceThe Simple Postoperative AKI Risk (SPARK) index is a prediction model for postoperative acute kidney injury (PO-AKI) in patients undergoing noncardiac surgery. External validation has not been performed.ObjectiveTo externally validate the SPARK index.Design, setting, and participantsThis single-center retrospective cohort study included adults who underwent noncardiac surgery under general anesthesia from 2007 to 2011. Those with obstetric or urological surgery, estimated glomerular filtration rate (eGFR) of less than 15 mL/min/1.73 m2, preoperative dialysis, or an expected surgical duration of less than 1 hour were excluded. The study was conducted at Nara Medical University Hospital. Data analysis was conducted from January to July 2021.ExposuresRisk factors for AKI included in SPARK index.Main outcomes and measuresPO-AKI, defined as an increase in serum creatinine of at least 0.3 mg/dL within 48 hours or 150% compared with preoperative baseline value or urine output of less than 0.5 mL/kg/h for at least 6 hours within 1 week after surgery, and critical AKI, defined as either AKI stage 2 or greater and/or any AKI connected to postoperative death or requiring kidney replacement therapy before discharge. The discrimination and calibration of the SPARK index were examined with area under the receiver operating characteristic curves (AUC) and calibration plots, respectively.ResultsAmong 5135 participants (2410 [46.9%] men), 303 (5.9%) developed PO-AKI, and 137 (2.7%) developed critical AKI. Compared with the SPARK cohort, participants in our cohort were older (median [IQR] age, 56 [44-66] years vs 63 [50-73] years), had lower baseline eGFR (median [IQR], 82.1 [71.4-95.1] mL/min/1.73 m2 vs 78.2 [65.6-92.2] mL/min/1.73 m2), and had a higher prevalence of comorbidities (eg, diabetes: 3956 of 51 041 [7.8%] vs 802 [15.6%]). The incidence of PO-AKI and critical AKI increased as the scores on the SPARK index increased. For example, 10 of 593 participants (1.7%) in SPARK class A, indicating lowest risk, experienced PO-AKI, while 53 of 332 (16.0%) in SPARK class D, indicating highest risk, experienced PO-AKI. However, AUCs for PO-AKI and critical AKI were 0.67 (95% CI, 0.63-0.70) and 0.62 (95% CI, 0.57-0.67), respectively, and the calibration was poor (PO-AKI: y = 0.24x + 3.28; R2 = 0.86; critical AKI: y = 0.20x + 2.08; R2 = 0.51). Older age, diabetes, expected surgical duration, emergency surgery, renin-angiotensin-aldosterone system blockade use, and hyponatremia were not associated with PO-AKI in our cohort, resulting in overestimation of the predicted probability of AKI in our cohort.Conclusions and relevanceIn this study, the incidence of PO-AKI increased as the scores on the SPARK index increased. However, the predicted probability might not be accurate in cohorts with older patients with more comorbidities.

Project description:Some patients will develop chronic kidney disease after a hospitalization with acute kidney injury; however, no risk-prediction tools have been developed to identify high-risk patients requiring follow-up.To derive and validate predictive models for progression of acute kidney injury to advanced chronic kidney disease.Data from 2 population-based cohorts of patients with a prehospitalization estimated glomerular filtration rate (eGFR) of more than 45 mL/min/1.73 m2 and who had survived hospitalization with acute kidney injury (defined by a serum creatinine increase during hospitalization > 0.3 mg/dL or > 50% of their prehospitalization baseline), were used to derive and validate multivariable prediction models. The risk models were derived from 9973 patients hospitalized in Alberta, Canada (April 2004-March 2014, with follow-up to March 2015). The risk models were externally validated with data from a cohort of 2761 patients hospitalized in Ontario, Canada (June 2004-March 2012, with follow-up to March 2013).Demographic, laboratory, and comorbidity variables measured prior to discharge.Advanced chronic kidney disease was defined by a sustained reduction in eGFR less than 30 mL/min/1.73 m2 for at least 3 months during the year after discharge. All participants were followed up for up to 1 year.The participants (mean [SD] age, 66 [15] years in the derivation and internal validation cohorts and 69 [11] years in the external validation cohort; 40%-43% women per cohort) had a mean (SD) baseline serum creatinine level of 1.0 (0.2) mg/dL and more than 20% had stage 2 or 3 acute kidney injury. Advanced chronic kidney disease developed in 408 (2.7%) of 9973 patients in the derivation cohort and 62 (2.2%) of 2761 patients in the external validation cohort. In the derivation cohort, 6 variables were independently associated with the outcome: older age, female sex, higher baseline serum creatinine value, albuminuria, greater severity of acute kidney injury, and higher serum creatinine value at discharge. In the external validation cohort, a multivariable model including these 6 variables had a C statistic of 0.81 (95% CI, 0.75-0.86) and improved discrimination and reclassification compared with reduced models that included age, sex, and discharge serum creatinine value alone (integrated discrimination improvement, 2.6%; 95% CI, 1.1%-4.0%; categorical net reclassification index, 13.5%; 95% CI, 1.9%-25.1%) or included age, sex, and acute kidney injury stage alone (integrated discrimination improvement, 8.0%; 95% CI, 5.1%-11.0%; categorical net reclassification index, 79.9%; 95% CI, 60.9%-98.9%).A multivariable model using routine laboratory data was able to predict advanced chronic kidney disease following hospitalization with acute kidney injury. The utility of this model in clinical care requires further research.

Project description:BackgroundAcute kidney injury (AKI) represents a frequent and grave complication associated with acute pancreatitis (AP), substantially elevating both mortality rates and the financial burden of hospitalization. The aim of our study is to construct a predictive model utilizing automated machine learning (AutoML) algorithms for the early prediction of AKI in patients with AP.MethodsWe retrospectively analyzed patients who were diagnosed with AP in our hospital from January 2017 to December 2021. These patients were randomly allocated into a training set and a validation set at a ratio of 7:3. To develop predictive models for each set, we employed the least absolute shrinkage and selection operator (LASSO) algorithm along with AutoML. A nomogram was developed based on multivariate logistic regression analysis outcomes. The model's efficacy was assessed using receiver operating characteristic (ROC) curves, calibration curves, and decision curve analysis (DCA). Additionally, the performance of the model constructed via AutoML was evaluated using decision curve analysis (DCA), feature importance, SHapley Additive exPlanations (SHAP) plots, and locally interpretable model-agnostic explanations (LIME).ResultsThis study incorporated a total of 437 patients who met the inclusion criteria. Out of these, 313 were assigned to the training cohort and 124 to the validation cohort. In the training and validation cohorts, AKI occurred in 68 (21.7%) and 29(23.4%) patients, respectively. Comparative analysis revealed that the AutoML models exhibited enhanced performance over traditional logistic regression (LR). Furthermore, the deep learning (DL) model demonstrated superior predictive accuracy, evidenced by an area under the ROC curve of 0.963 in the training set and 0.830 in the validation set, surpassing other comparative models. The key variables identified as significant in the DL model within the training dataset included creatinine (Cr), urea (Urea), international normalized ratio (INR), etiology, smoking, alanine aminotransferase (ALT), hypertension, prothrombin time (PT), lactate dehydrogenase (LDH), and diabetes.ConclusionThe AutoML model, utilizing DL algorithm, offers considerable clinical significance in the early detection of AKI among patients with AP.