Dataset Information

Validation of Risk Prediction Models to Inform Clinical Decisions After Acute Kidney Injury.

ABSTRACT:

Rationale & objective

There is limited evidence to guide follow-up after acute kidney injury (AKI). Knowledge gaps include which patients to prioritize, at what time point, and for mitigation of which outcomes. In this study, we sought to compare the net benefit of risk model-based clinical decisions following AKI.

Study design

External validation of 2 risk models of AKI outcomes: the Grampian -Aberdeen (United Kingdom) AKI readmissions model and the Alberta (Canada) kidney disease risk model of chronic kidney disease (CKD) glomerular (G) filtration rate categories 4 and 5 (CKD G4 and G5). Process mining to delineate existing care pathways.

Setting & participants

Validation was based on data from adult hospital survivors of AKI from Grampian, 2011-2013.

Predictors

KDIGO-based measures of AKI severity and comorbidities specified in the original models.

Outcomes

Death or readmission within 90 days for all hospital survivors. Progression to new CKD G4-G5 for patients surviving at least 90 days after AKI.

Analytical approach

Decision curve analysis to assess the "net benefit" of use of risk models to guide clinical care compared to alternative approaches (eg, prioritizing all AKI, severe AKI, or only those without kidney recovery).

Results

26,575 of 105,461 hospital survivors in Grampian (mean age, 60.9 ± 19.8 [SD] years) were included for validation of the death or readmission model, and 9,382 patients (mean age, 60.9 ± 19.8 years) for the CKD G4-G5 model. Both models discriminated well (area under the curve [AUC], 0.77 and 0.86, respectively). Decision curve analysis showed greater net benefit for follow up of all AKI than only severe AKI in most cases. Both original and refitted models provided net benefit superior to any other decision strategy. In process mining of all hospital discharges, 41% of readmissions and deaths occurred among people recovering after AKI. 1,464 of 3,776 people (39%) readmitted after AKI had received no intervening monitoring.

Limitations

Both original models overstated risks, indicating a need for regular updating.

Conclusions

Follow up after AKI has potential net benefit for preempting readmissions, death, and subsequent CKD progression. Decisions could be improved by using risk models and by focusing on AKI across a full spectrum of severity. The current lack of monitoring among many with poor outcomes indicates possible opportunities for implementation of decision support.

SUBMITTER: Sawhney S

PROVIDER: S-EPMC8234511 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ImportanceSome patients will develop chronic kidney disease after a hospitalization with acute kidney injury; however, no risk-prediction tools have been developed to identify high-risk patients requiring follow-up.ObjectiveTo derive and validate predictive models for progression of acute kidney injury to advanced chronic kidney disease.Design, setting, and participantsData from 2 population-based cohorts of patients with a prehospitalization estimated glomerular filtration rate (eGFR) of more than 45 mL/min/1.73 m2 and who had survived hospitalization with acute kidney injury (defined by a serum creatinine increase during hospitalization > 0.3 mg/dL or > 50% of their prehospitalization baseline), were used to derive and validate multivariable prediction models. The risk models were derived from 9973 patients hospitalized in Alberta, Canada (April 2004-March 2014, with follow-up to March 2015). The risk models were externally validated with data from a cohort of 2761 patients hospitalized in Ontario, Canada (June 2004-March 2012, with follow-up to March 2013).ExposuresDemographic, laboratory, and comorbidity variables measured prior to discharge.Main outcomes and measuresAdvanced chronic kidney disease was defined by a sustained reduction in eGFR less than 30 mL/min/1.73 m2 for at least 3 months during the year after discharge. All participants were followed up for up to 1 year.ResultsThe participants (mean [SD] age, 66 [15] years in the derivation and internal validation cohorts and 69 [11] years in the external validation cohort; 40%-43% women per cohort) had a mean (SD) baseline serum creatinine level of 1.0 (0.2) mg/dL and more than 20% had stage 2 or 3 acute kidney injury. Advanced chronic kidney disease developed in 408 (2.7%) of 9973 patients in the derivation cohort and 62 (2.2%) of 2761 patients in the external validation cohort. In the derivation cohort, 6 variables were independently associated with the outcome: older age, female sex, higher baseline serum creatinine value, albuminuria, greater severity of acute kidney injury, and higher serum creatinine value at discharge. In the external validation cohort, a multivariable model including these 6 variables had a C statistic of 0.81 (95% CI, 0.75-0.86) and improved discrimination and reclassification compared with reduced models that included age, sex, and discharge serum creatinine value alone (integrated discrimination improvement, 2.6%; 95% CI, 1.1%-4.0%; categorical net reclassification index, 13.5%; 95% CI, 1.9%-25.1%) or included age, sex, and acute kidney injury stage alone (integrated discrimination improvement, 8.0%; 95% CI, 5.1%-11.0%; categorical net reclassification index, 79.9%; 95% CI, 60.9%-98.9%).Conclusions and relevanceA multivariable model using routine laboratory data was able to predict advanced chronic kidney disease following hospitalization with acute kidney injury. The utility of this model in clinical care requires further research.

Project description:Acute kidney injury (AKI) is a frequent complication of liver transplantation and is associated with increased mortality. We identified the incidence and modifiable risk factors for AKI after living-donor liver transplantation (LDLT) and constructed risk scoring models for AKI prediction. We retrospectively reviewed 538 cases of LDLT. Multivariate logistic regression analysis was used to evaluate risk factors for the prediction of AKI as defined by the RIFLE criteria (RIFLE = risk, injury, failure, loss, end stage). Three risk scoring models were developed in the retrospective cohort by including all variables that were significant in univariate analysis, or variables that were significant in multivariate analysis by backward or forward stepwise variable selection. The risk models were validated by way of cross-validation. The incidence of AKI was 27.3% (147/538) and 6.3% (34/538) required postoperative renal replacement therapy. Independent risk factors for AKI by multivariate analysis of forward stepwise variable selection included: body-mass index >27.5 kg/m2 [odds ratio (OR) 2.46, 95% confidence interval (CI) 1.32-4.55], serum albumin <3.5 mg/dl (OR 1.76, 95%CI 1.05-2.94), MELD (model for end-stage liver disease) score >20 (OR 2.01, 95%CI 1.17-3.44), operation time >600 min (OR 1.81, 95%CI 1.07-3.06), warm ischemic time >40 min (OR 2.61, 95%CI 1.55-4.38), postreperfusion syndrome (OR 2.96, 95%CI 1.55-4.38), mean blood glucose during the day of surgery >150 mg/dl (OR 1.66, 95%CI 1.01-2.70), cryoprecipitate > 6 units (OR 4.96, 95%CI 2.84-8.64), blood loss/body weight >60 ml/kg (OR 4.05, 95%CI 2.28-7.21), and calcineurin inhibitor use without combined mycophenolate mofetil (OR 1.87, 95%CI 1.14-3.06). Our risk models performed better than did a previously reported score by Utsumi et al. in our study cohort. Doses of calcineurin inhibitor should be reduced by combined use of mycophenolate mofetil to decrease postoperative AKI. Prospective randomized trials are required to address whether artificial modification of hypoalbuminemia, hyperglycemia and postreperfusion syndrome would decrease postoperative AKI in LDLT.

Project description:IntroductionNumerous prediction models have been developed to support treatment-related decisions for breast cancer patients. External validation, a prerequisite for implementation in clinical practice, has been performed for only a few models. This study aims to externally validate published clinical prediction models using population-based Dutch data.MethodsPatient-, tumor- and treatment-related data were derived from the Netherlands Cancer Registry (NCR). Model performance was assessed using the area under the receiver operating characteristic curve (AUC), scaled Brier score, and model calibration. Net benefit across applicable risk thresholds was evaluated with decision curve analysis.ResultsAfter assessing 922 models, 87 (9%) were included for validation. Models were excluded due to an incomplete model description (n = 262 (28%)), lack of required data (n = 521 (57%)), previously validated or developed with NCR data (n = 45 (5%)), or the associated NCR sample size was insufficient (n = 7 (1%)). The included models predicted survival (33 (38%) overall, 27 (31%) breast cancer-specific, and 3 (3%) other cause-specific), locoregional recurrence (n = 7 (8%)), disease free survival (n = 7 (8%)), metastases (n = 5 (6%)), lymph node involvement (n = 3 (3%)), pathologic complete response (n = 1 (1%)), and surgical margins (n = 1 (1%)). Seven models (8%) showed poor (AUC<0.6), 39 (45%) moderate (AUC:0.6-0.7), 38 (46%) good (AUC:0.7-0.9), and 3 (3%) excellent (AUC≥0.9) discrimination. Using the scaled Brier score, worse performance than an uninformative model was found in 34 (39%) models.ConclusionComprehensive registry data supports broad validation of published prediction models. Model performance varies considerably in new patient populations, affirming the importance of external validation studies before applying models in clinical practice. Well performing models could be clinically useful in a Dutch setting after careful impact evaluation.

Project description:BackgroundThere are many Machine Learning (ML) models which predict acute kidney injury (AKI) for hospitalised patients. While a primary goal of these models is to support clinical decision-making, the adoption of inconsistent methods of estimating baseline serum creatinine (sCr) may result in a poor understanding of these models' effectiveness in clinical practice. Until now, the performance of such models with different baselines has not been compared on a single dataset. Additionally, AKI prediction models are known to have a high rate of false positive (FP) events regardless of baseline methods. This warrants further exploration of FP events to provide insight into potential underlying reasons.ObjectiveThe first aim of this study was to assess the variance in performance of ML models using three methods of baseline sCr on a retrospective dataset. The second aim was to conduct an error analysis to gain insight into the underlying factors contributing to FP events.Materials and methodsThe Intensive Care Unit (ICU) patients of the Medical Information Mart for Intensive Care (MIMIC)-IV dataset was used with the KDIGO (Kidney Disease Improving Global Outcome) definition to identify AKI episodes. Three different methods of estimating baseline sCr were defined as (1) the minimum sCr, (2) the Modification of Diet in Renal Disease (MDRD) equation and the minimum sCr and (3) the MDRD equation and the mean of preadmission sCr. For the first aim of this study, a suite of ML models was developed for each baseline and the performance of the models was assessed. An analysis of variance was performed to assess the significant difference between eXtreme Gradient Boosting (XGB) models across all baselines. To address the second aim, Explainable AI (XAI) methods were used to analyse the XGB errors with Baseline 3.ResultsRegarding the first aim, we observed variances in discriminative metrics and calibration errors of ML models when different baseline methods were adopted. Using Baseline 1 resulted in a 14% reduction in the f1 score for both Baseline 2 and Baseline 3. There was no significant difference observed in the results between Baseline 2 and Baseline 3. For the second aim, the FP cohort was analysed using the XAI methods which led to relabelling data with the mean of sCr in 180 to 0 days pre-ICU as the preferred sCr baseline method. The XGB model using this relabelled data achieved an AUC of 0.85, recall of 0.63, precision of 0.54 and f1 score of 0.58. The cohort size was 31,586 admissions, of which 5,473 (17.32%) had AKI.ConclusionIn the absence of a widely accepted method of baseline sCr, AKI prediction studies need to consider the impact of different baseline methods on the effectiveness of ML models and their potential implications in real-world implementations. The utilisation of XAI methods can be effective in providing insight into the occurrence of prediction errors. This can potentially augment the success rate of ML implementation in routine care.

Project description:ImportanceThe Simple Postoperative AKI Risk (SPARK) index is a prediction model for postoperative acute kidney injury (PO-AKI) in patients undergoing noncardiac surgery. External validation has not been performed.ObjectiveTo externally validate the SPARK index.Design, setting, and participantsThis single-center retrospective cohort study included adults who underwent noncardiac surgery under general anesthesia from 2007 to 2011. Those with obstetric or urological surgery, estimated glomerular filtration rate (eGFR) of less than 15 mL/min/1.73 m2, preoperative dialysis, or an expected surgical duration of less than 1 hour were excluded. The study was conducted at Nara Medical University Hospital. Data analysis was conducted from January to July 2021.ExposuresRisk factors for AKI included in SPARK index.Main outcomes and measuresPO-AKI, defined as an increase in serum creatinine of at least 0.3 mg/dL within 48 hours or 150% compared with preoperative baseline value or urine output of less than 0.5 mL/kg/h for at least 6 hours within 1 week after surgery, and critical AKI, defined as either AKI stage 2 or greater and/or any AKI connected to postoperative death or requiring kidney replacement therapy before discharge. The discrimination and calibration of the SPARK index were examined with area under the receiver operating characteristic curves (AUC) and calibration plots, respectively.ResultsAmong 5135 participants (2410 [46.9%] men), 303 (5.9%) developed PO-AKI, and 137 (2.7%) developed critical AKI. Compared with the SPARK cohort, participants in our cohort were older (median [IQR] age, 56 [44-66] years vs 63 [50-73] years), had lower baseline eGFR (median [IQR], 82.1 [71.4-95.1] mL/min/1.73 m2 vs 78.2 [65.6-92.2] mL/min/1.73 m2), and had a higher prevalence of comorbidities (eg, diabetes: 3956 of 51 041 [7.8%] vs 802 [15.6%]). The incidence of PO-AKI and critical AKI increased as the scores on the SPARK index increased. For example, 10 of 593 participants (1.7%) in SPARK class A, indicating lowest risk, experienced PO-AKI, while 53 of 332 (16.0%) in SPARK class D, indicating highest risk, experienced PO-AKI. However, AUCs for PO-AKI and critical AKI were 0.67 (95% CI, 0.63-0.70) and 0.62 (95% CI, 0.57-0.67), respectively, and the calibration was poor (PO-AKI: y = 0.24x + 3.28; R2 = 0.86; critical AKI: y = 0.20x + 2.08; R2 = 0.51). Older age, diabetes, expected surgical duration, emergency surgery, renin-angiotensin-aldosterone system blockade use, and hyponatremia were not associated with PO-AKI in our cohort, resulting in overestimation of the predicted probability of AKI in our cohort.Conclusions and relevanceIn this study, the incidence of PO-AKI increased as the scores on the SPARK index increased. However, the predicted probability might not be accurate in cohorts with older patients with more comorbidities.